Blog

Field Normalization

Field normalization helps you resolve data quality issues in ServiceNow.  

Records for things like devices and companies are brought into the ServiceNow platform by manual entry, imports, and Discovery. Depending on how it is introduced into the database, a field value might appear in several different forms. For example, the CPU Type field on a computer CI form might display any of the following names for the same type of CPU, depending on the source of the entry:

The version of field normalization included from ServiceNow is free.  It includes two different types of normalization and a couple of examples of how to use it.  If you have specific or limited amount of data to normalize, this application can help you out greatly.

For large amounts of normalization or low-maintenance normalization is where vendors like BDNA and Eracent are selling normalization "technopedia" data.  This costs more than free of course.

Activate

To use Field Normalization, activate the "Field Normalization" plugin.  

Please note this doesn't activate all the example data normalization and transformation jobs, it just activates the plugin.  You can activate the normalization jobs as you see fit.

I would suggest always trying out normalization in a development instance first, as it is very powerful and can cause a mess if you don't test first.

TYPES

Field Normalization includes two types: normalization and transformation.

  • Normalization forces the platform to convert different forms of the same field value to a single, accepted value automatically. By forcing a field to use a simple, recognizable description for multiple variations of the same thing, normalization can eliminate duplicate records and make searches easier. In addition to reconciling different forms of the same value in fields, normalization can be configured to adjust queries automatically to return normalized results.
  • Transformation enables an administrator to transform raw field input into standardized values that are more meaningful to an organization. An example of a standardized value might be to round RAM size in configuration items to a whole number, such as 4000 MB instead of 4112 MB. Transformations are controlled by parameters and conditions and can be configured to return transformed values in queries.

NORMALIZATION EXAMPLES

EXAMPLE 1: MANUFACTURER

HP has various different ways they can be named in ServiceNow due to various imports and discovery scans.

  • HP
  • HP Inc.
  • Hewlett-Packard
  • Compaq

All these different versions can make reporting difficult.  Field Normalization takes all these companies and can normalizes them to one: "HP".  

EXAMPLE 2: OPERATING SYSTEM

When you import configuration items from Discovery, the operating system will be slightly different depending on what is stored in that computer's information.

  • Microsoft Windows 7 Professional
  • Microsoft Windows 7 Pro
  • Microsoft Windows Professional 7 
  • Windows 7 Pro
  • Windows® 7 Pro
  • Windows ® 7 Pro

Those slight differences make reporting and license reconciliation difficult.  Field normalization "normalizes that info into a common name like "Microsoft Windows 7 Professional".

TRANSFORMATION EXAMPLES

EXAMPLE 1: DISK SPACE

Disk space from discovery sources can be formatted in all kinds of odd increments

  • Under 10GB, round half up
  • Over 10GB, round half up
  • Over 100GB, round half up
  • Over a Terabyte, round half up

Using this method, you get nice rounded totals for disk space.  Instead of 9.8GB, you get 10GB.  Instead of 8.9GB, you get 10GB.  It makes reporting on disk space so much nicer.

HOW IT WORKS: PROCESS

Step 1: CREATE NORMALIZATION RECORD

1. Navigate to Field Normalization > Configurations > Normalizations.
2. Click New.
3. Create a normalization record.  I suggest using one of the example records as a reference.

Field | Input value
Name | Unique name for this normalization record. This value is for reference only and is not used in any processing.
Table | Select the ServiceNow table for the field being normalized.
Field | Select the field to normalize.
Mode | The three available modes are Off, Test, and Active. All normalization records are created in the test mode by default. If you are planning to select aliases for your normal values, change the Mode to Active. If you intend to normalize the field using rules, be sure to leave this record in the Test mode. To disable this normalization, switch the mode to Off.
Normalize query | Select this check box to apply the field value normalized by this record to all queries involving this field. Queries formed with the raw (original) field value return records displaying the normal value. Queries issued by a script using the conditions equals or not equals return normalized values. See Normalized Queries for details.
Coalesce each normal | Select this check box to reset all references in the database to records containing an alias field value to a single record using a normal value. See Coalesce Normal Values for details.
Raw field | Select the field to use to display the original (non-normalized) input values on a form in which a field value has been normalized. For the selection to appear in the drop-down list, add a custom field to the form for the table selected. For instructions on adding a field for raw data, see Creating a Raw Field.

4. Click Save
5. Click "Collect Pending Values" if the Pending Value Collection job is not added.  The "Pending Value Collection job" collects all the possible values of the field you can normalize.

STEP 2: Create Normal Values

A  normal value is a simplified, generic value for a field that replaces all the possible variants of that value that exist in the database.

A good example of this is the out-of-the-box CPU Type example.  By setting up some normal values, it normalizes all the different versions of the Xeon cpu to be just "Xeon".

STEP 3A: Create Alias

Aliases are the variants of a field value in the instance that will be replaced by the normal value.

For the Xeon Normal value, you would select all the Pending values that would match to that normal value.

This is intended for smaller-scale normalization.  If you need to normalize a large amount of data, this may be very maintenance intensive.

STEP 3B: SETUP RULES

Rules are a lot more maintenance-free than normal values.  I would suggest using these instead of alias.

Some example rules:

  • Intel Xeon: CPU type matches pattern *Intel*Xeon*. This rule normalizes all variants in which Intel precedes Xeon, including Intel Xeon, Intel(R) Xeon(TM) CPU 2.80GHz, and Intel(R) Xeon(TM) CPU 3.00GHz.
  • Xeon: CPU type contains Xeon
  • L3350: CPU type contains L3350
  • E3350: CPU type contains E3350

Rules are dynamic and less maintenance intensive than an alias as you may notice.

Be careful to not have a blank expression in your rules.  When you run your data job, it will set all the fields to that rule.

Step 4: Optional Raw Field 

A raw field is a custom field created by an administrator to show the original (raw) input in a field on a form after it has been normalized or transformed.

This is helpful for debugging and other reasons.

STEP 5: ACTIVATE

Transformations have an optional Test Transforms feature, Normalizations don't have that.  Either way, I always suggest running normalizations or transformations in a development instance first.

When you are ready to run your job, flip the Mode to "Active" and click save.  You'll notice new data jobs in the Data jobs tab.

Only flip to active when you are sure you are ready.

STEP 6: RUN Data Jobs

Field Normalization creates four data jobs during the normalization process to collect or change data.

All jobs except Pending value collection are executed manually. Data jobs have a roll back feature that allows you to undo normalization at different stages before committing changes that affect the entire database.

  • Pending value collection: This data job is created when a field normalization record is submitted. This job runs automatically to collect all the values from the database for the field to be normalized. This data job does not modify the database.
  • Normal value change: This data job is created when an existing normal value that is used to normalize a field is changed. When the job is run, the platform updates all the normalized fields in the database that use the new normal value.
  • Alias application: This data job is created when an alias is created. When the job is run, all field values in the database containing the alias are normalized.
  • Rule application: This data job is created when a rule is created. When the job is run, all field values in the database that match this rule are normalized.
  • Coalesce to normal: This data job coalesces data from existing records containing a normalized field value into a single record that uses a normal value.

You may have to go into these data jobs and click Start if they don't run automatically.

STEP 7: NORMALIZATION IN ACTION

When all the data jobs finished, you'll have normalized data.

Normalized fields are marked with an icon that links to different targets, depending on the user's role.

Users with the normalizer role can click the icon to access the normalization configuration record for that field directly. When a user without the normalizer role clicks the link, a help page appears that provides help on the form. A preference, called Restrict to roles, enables an administrator to define the roles that are permitted to see the icon. If no roles are specified, then the icon is visible to all roles.

STEP 7: Rollback

You can rollback a job if you mess up, however I don't recommend this.  Be careful the first time around and don't rely on this.  Always test in a development instance first!

More Information

For more information on Field Normalization, check out the Product Documentation on Field Normalization.