At Xeneta, maintaining the high quality of our ocean and air freight data is always a priority. Our rate management team (RMT) even has its own motto - “Speed first, quality always.”
To ensure that we deliver the best possible quality of data, we have set up multiple checks and validations at different data processing stages.
Read more how about Xeneta’s data qualification processes below.
Stage 1: Pre-upload Data Assessment and Cleaning
We thoroughly check all information received from the customer when processing the freight rates, including the rate sheets, notes, comments, and any additional files.
Furthermore, we have a set of requirements (aka Data guidelines) to review whether the data received from the customer is complete or if we need to request additional clarifications.
At this stage, we pay special attention to:
- Validity of the rates to ensure that rates are compared within the correct dates and market (short/long);
- Locations, to make sure we benchmark the prices of accurate and correct ports and don’t include pre-/on-carriage costs;
- Modes of transport, to exclude, e.g., high-priced truckage that is generally incomparable to feeder/rail transportation (where applicable);
- rates' commodities to make sure that high value and extra handling costs do not distort the benchmarks;
- Equipment types the rates belong to, to ensure we are comparing the rates within the correct size and type of container;
- Price levels, to check for typos and other similar kinds of errors, we pay special attention to the prices that seem unnaturally high or low, negative prices, zero rates, etc.;
- Surcharges, to bring all the rates to the same standard and make sure no unnecessary charges are included, as well as that the surcharges are applied to the correct rates, with the correct validity and the correct amount;
- Currencies for both base freight and surcharges so that the total dollar price of each rate is correct;
- Validity of the surcharges, so they are timely updated (e.g., fuel charges or PSS) and the correct amount is applied for a specific time period;
- Volumes so that the customer’s spend overview and other volume-related information is precise;
- Service provider, so a customer has an overview of their rates, and we have accurate data behind our carrier spread;
- Contract numbers, so it’s easy for both customers and us to track the development of a particular rate;
Additionally, for air freight rates, we ensure that we are aligned on the service level and weight bracket and measure the unit of each freight rate.
If there are any doubts regarding the above-mentioned points, we double-check with the customer before we proceed with ingesting the rates into our platform. Many of these checks are automated, while some particularly tricky ones are manual. After finalizing the processing of a rate sheet, we always look over the rates again before uploading them into Xeneta.
Stage 2: Data Upload & Validation
We have an additional validation stage during the uploading process of the rates into the platform. All the uploaded data is checked against a defined set of rules during this stage. At this step, we mainly concentrate on the items mentioned above, double-checking the processing results or looking at the same point from a different angle.
If rates violate any of the rules mentioned above, they are escalated to the analyst team for further investigation or correction. In case the issue with a particular rate can’t be solved, they are discarded.
Stage 3: Outlier detection
At the end of the processing, all the rates go through the outlier detection phase. In simple terms, we re-examine the rates that seem outliers from a statistical point of view but have no data quality issues.
This process ensures that the market metrics are not distorted by one rate or a small number of rates significantly lower or higher than the rest of the prices. (A datapoint that falls out of 3 standard deviations from the mean is usually considered an outlier.)
NOTE: Outliers are revised periodically (both automatically and manually) for the best fit and to ensure we provide as accurate market metrics as possible.
Stage 4: Final Validation
After the rates are already in a customer’s account, we do a final review to ensure all the rates are uploaded and appropriately calculated. We also check if the rates are accurately displayed in a customer’s account. There are generally no disruptions during any of the processing stages.
Bonus Stage: Regular Data Quality Maintenance
We have a set of tools that help us guarantee our data quality on a higher level. Our RMT team also holds regular sessions to focus on ensuring the consistency and correctness of our data.
Moreover, we encourage all our customers to report anything that seems suspicious on the platform. We commit to promptly investigating every such request and ensuring that any possible issue is solved as quickly as possible.
Want to learn more?
Schedule a personalized demo of the Xeneta platform tailored to match your container shipping and air cargo procurement strategy.
