ACCEPTING MAVEN OUTPUT


Maven reports are designed to be configurable at the company or organization level. The goal is to reduce the time and effort compared to traditional appraisals or evaluations. Depending on configuration, security settings, and output template, the final report is still an appraisal or evaluation. However, Maven can also be used in an automated manner to produce Automated Valuation Model (AVM) output.


Organizations using Maven need to consider a variety of risk factors to determine what type of valuation model and report is appropriate.  AgWare is not an expert in risk compliance. This document is intended to collect the input received from our clients and consultants concerning models and output and to highlight why we believe Maven can be used for some portion of your valuation needs.


Most of the literature on AVMs is centered around mass appraisal questions, usually related to tax valuations. The stereotypical AVM is a formula developed from applying linear regression to a sales database. The formula generated from the regression is then applied to multiple new subject properties.


In contrast, Maven is designed to apply a model to an individual subject property instead of a group of subjects at once. Because of this, AgWare refers to the process as MAV or Model Assisted Valuation.


MODEL TYPE CONSIDERATIONS


At the beginning of development, AgWare considered three model types: Regression, Range of Value, and Sales Comparison Approach. AgWare is positioned to the development process incrementally, i.e.,


    • Version 1 of Maven only contains the Sales Comparison Approach
    • We will continue to evaluate other models and may add them to future versions. 


During the project startup, AgWare contacted several Farm Credit associations and requested sales data to use in our testing. The request was for a group of sales, over several years, that represented a market for the type of properties they believed would work in an alternate valuation product. We ran multiple tests against the data to determine what approach provided the most consistent, understandable output.


The regression model, although traditionally used in automated valuations products, was not as strong or reliable as the sales comparison model. We found it difficult to identify and properly use the independent variables needed to run the regression. Once a formula was developed we found it hard to explain the output.  We also had trouble determining if the output value was reasonable. The regression formula is normally static for some time period. To get around this issue, we have seen products that recalculate the regression formula, based on current data, for each subject property. We think that puts too much burden on appraisers and evaluators to understand the details of the regression process. We may still build a regression model in Maven but that will be determined by the success of the other models and customer requests.


A range of value model was also considered. This type of valuation is currently used by a number of our customers. A range of value model will be built into the next version Maven. We know this model will be acceptable to risk management since it is already in use. We believe that our process in Maven can help reduce the time needed to create range of value reports. The downside to range of value models is they require upfront work from appraisers to conduct market studies and establish and maintain ranges.


SALES COMPARISON MODEL


The sales comparison approach is the first model available in Maven for several reasons. Note that in the following discussion I will be referring to appraisers and appraisals. For the sake of brevity, I will not address “evaluator(s)” in each case, but it is implied.


First, when a model is run by a specific individual, for a single subject property, we believe the output is the person’s instead of the models. That is to say, if an appraiser enters the subject property description and presses the run model button, the output is an appraisal and must be signed by the appraiser. This means the appraiser must be able to understand and agree with the work done by the model. Appraisers are trained to complete three standard approaches to value: Sales Comparison Approach, Cost Approach, and the Income Approach. Having the model generate a completed Sales Comparison Approach gives the appraiser something they can read and logically understand. Allowing appraisers to optionally make changes to the output ensures the final number is their own opinion of value. We interviewed organizations regarding appraisals they were hoping to complete in Maven. We looked at the approaches completed and any adjustments made in current reports. There was broad consensus, for the property types in question, a sales comparison approach was sufficient for a reliable appraisal. Additionally, we found that for these properties the majority of appraisals did not make adjustments beyond a land mix adjustment and time (market condition) adjustment.


The primary goal of the Maven Sales Comparison Model is to automate the process of selecting comparable sales by mirroring appraisers’ existing processes. If enough sales can be selected, and if they meet a consistency test, the data can be processed through a template engine to produce the final appraisal report. If the subject cannot be valued due to lack of data, or lack of consistency in the data, Maven can reject the report. An appraiser can then complete a ClickForms report using the full power of the AgWare tool-set.


When correctly configured, the final value in a Maven report should be a similar number an appraiser would have concluded in a ClickForms appraisal report. The normal appraisal process involves opening a ClickForms template, keying in subject data, searching a sales database for comparable sales, importing the most comparable sales into ClickForms, and finally keying in the final opinion of value. In Maven, the subject data is keyed in and then the software performs the remaining steps automatically. If the model successfully selects the same sales an appraiser would have selected, no further work from the appraiser is needed. If the appraiser does not believe the sales are appropriate, they can adjust the selected sales "list" to match what they would have manually selected.


When changes are made to the selected sales, a "log of changes and reasons" are collected and saved by Maven. This allows models to be adjusted over time. If the wrong sales are being selected by the model, additional selection factors can be added to better match the appraisal process in that market area. The "change log" also provides an audit point for reviews to verify that the output is not being inappropriately adjusted. Because the output reports are appraisals that show selected sales, adjustments, and conclusions they can also be examined under a normal appraisal review process.


A critical component to this process is an up to date sales database with verified, consistently entered and allocated sales. We do not believe our process will work using only publicly available data. Agricultural properties, even non-complex properties, contain multiple components that must be consistently allocated. The allocation process between multiple land types and improvements allows a mathematical calculation referred to as the land and/or building mix adjustments to be automatically made for a subject property. This calculation can only be made if the sales are consistently allocated.


In addition to the allocation, any selection criteria need to be explicitly tracked. Organizations may need they need to adjust how sales data is entered so all needed selection data is individually classified. The system cannot select comparable sales based on data that is only provided in the comments.


ACCEPTABLE MODEL STATISTICS


When a building a model in Maven, a testing procedure is run to determine how well the model performs. A common question we receive is, how good does the model need to be?   Any appraisal process, including narrative reports will vary from actual sale prices. Our goal is to have a model perform close to the same range as appraisals done without Maven. This does not mean that comparing a completed report with a Maven test will always show the same result. The overall spread is what should be similar. Unfortunately studies are not typically performed on traditional appraisals, so it is impossible to compare accuracy statics in a head to head manner.


Based on our testing and research we suggest that a good model will show the Average and Median Accuracy between 90% and 110%, a Coefficient of Dispersion less than 20, and that 70% of valued subjects are within 20% of the anticipated value.


We also suggest looking closely at the most overvalued subjects in the testing set. Overvalued subjects have the most risk. Our test data has indicated that many times these high results are using sales as subjects that were not true arm’s length transactions. If that is not the case, or if the distribution of values is skewed toward the higher percentages the model may need to be adjusted to ensure subjects are not being overvalued.


The override feature provides the final safety valve to ensure that values are reasonable. The model testing will only report the default value produced by the value. In the actual valuation of an individual subject, the organization can choose to leave the final opinion of value in the hands of the appraiser or evaluator.