Data Management

a5

Data Processing

After the input of the source data, the next phase is the preprocessing of data, manipulation of the missing values and data visualization. Often, data will require pre-processing before they can be used in actual analysis. By default, this process is automated, and the appropriate algorithm(s) are selected based on the characteristics of the dataset used and the desired result; however, various data pre-processing options are available in case manual configuration is desired. The training and test sets allows for various operations to be performed on source data.

Often, data sets contain missing values. PredMine provides options for addressing the issue of missing values . The available options are “autofill”, “keep”, “remove”, and “stop”.

– Autofill uses appropriate statistical techniques (missing value analysis) to compute the missing values automatically.
– Keep retains the missing values (i.e. no value is assigned).
– Remove excludes cases with missing values from the model building process.
– Stop stops the execution and an informative message is displayed on the screen.

a6

Data Preparation

The Data Preparation step provides the following options:

1. Split Data to create Test Data. If no test set has been imported in the previous window, the user has the option of splitting the existing set in two parts, so that one of them will act as the test set, based on which the training set will be evaluated. Additionally, there is the option of dividing the data into more than two subsets so as to apply cross-validation techniques as a means of preventing overfitting. User-controlled parameters include the following:

– Number of segments. The default value is 2 (one segment to function as a train set, another one to function as the train set). If cross-validation is desired, this value can be increased.

– Relative segment size. By default, the train set consists of 70% of the entire dataset (determined at random), whereas the remaining 30% forms the test set. These percentages can be changed – e.g. they can be set to 50%, resulting in train and test sets of equal size. If more than two segments are requested (see previous option), the user may supply a value for each segment. These values can be automatically computed (e.g. the user selects the option “Equal size”).

2. Explore Data. Descriptive statistics (e.g. frequency tables, mean, median, mode, standard deviation, skewness, kurtosis, minimum and maximum values, range, etc.) are displayed in tabular form. Additionally, tests of normality (e.g. Kolmogorov-Smirnov / Lilliefors, Shapiro – Wilk, Pearson’s chi-square, Anderson – Darling) can be performed on the data. The user selects the appropriate statistical analysis tasks to be performed.

Top