Skip to main content

Building Predictive AI Models When Synthetic Data Is All You Have

Taniya Ghosh
Nov 19 - 5 min read
Building Predictive AI Models When Synthetic Data Is All You Have featured image

In our Engineering Energizers Q&A series, we highlight the engineering minds driving innovation across Salesforce. Today, we spotlight Taniya Ghosh, Senior Manager of Software Engineering, whose team faced a brutal engineering contradiction: build predictive AI models that warn of escalating incidents before they had real customer incidents to learn from. Synthetic training data gave them a starting point, but successful tests could not prove the models would predict SLA breaches and major incidents in real customer environments.

Explore how the team generated realistic synthetic training scenarios, validated predictions when accuracy alone could mislead, and connected Data 360’s ingestion, transformation, training, and scoring capabilities into reusable customer-specific pipelines.

What is your team’s mission, and why was making IT service management predictive so difficult?

The team had to warn service managers before an incident breached its service-level agreement (SLA) or became a major problem, without requiring customers to hire data scientists. Yet a model that worked for one customer could fail for another, and the team could not train across customers’ data. The challenge was making predictive modeling reusable while keeping every customer’s learning separate.

The mission was to move IT service management (ITSM) from reactive firefighting to predictive intervention. The team built three Predictive Model Templates: SLA Breach Prediction, Major Incident Prediction, and Change Risk Score. Earlier signals could help managers try to prevent escalation, or prepare to handle it, with explanations showing why an incident needed attention. Through Data 360, the team brought those signals into the incident-management workflow. Managers could see a risk score and its leading predictors on the incident record, giving them context for deciding what to investigate.

The work also supported Salesforce’s positioning and demos for its recognition as one of two Leaders in the Gartner Magic Quadrant for AI Applications in ITSM. SLA Breach and Major Incident Prediction supported major incident detection; Change Risk Score supported intelligent risk advisory, demonstrating Agentforce IT Service’s machine-learning depth.

What obstacles did you face generating synthetic training data without introducing unrealistic incident patterns?

Synthetic records could provide a development dataset without providing useful predictive patterns. The simulated Customer Zero data was insufficient, and adding unrealistic variations could leave the team evaluating features with little relevance to the incidents customers would encounter.

To ground synthetic data generation in ITSM scenarios, the team asked product managers for sample data covering as many real-world situations as possible. The team expanded those samples by varying features and used open-source LLMs to generate additional records within product managers’ guardrails. This produced synthetic training data for evaluating feature relevance and constructing test scenarios before customers could train the templates on their own Data 360 records. The team then needed to check whether predictions responded sensibly when a risk factor changed.


Why was predictive model validation so challenging when high accuracy could conceal a useless model?

SLA breaches were rare, making aggregate accuracy potentially misleading. As an illustrative example, a naive model could appear 95% accurate and still be useless for identifying breaches. Validation needed to examine whether predictions responded appropriately to risk factors and whether their explanations made sense to service managers.

The team validated behaviours across a range of scenarios, each isolating one factor. For example, in one scenario, they held every incident feature fixed and varied only the owners, which were ones who had missed 10% of past milestones and ones who had missed none. The incident associated with the owner with higher historical miss rate should receive a higher SLA breach score. That provided a specific expected response to check instead of relying solely on an aggregate accuracy number. Alongside the predicted probability on a 0 to 100 scale, the team examined the top three predictors to see whether they matched factors an experienced service manager would flag. The score had to move in the expected direction, and the explanation had to identify relevant reasons.

Statistical metrics remained part of development, but these comparisons allowed the team to inspect predictive behavior directly. They validated the tested scenarios; they did not establish production accuracy across customers using their own Data 360 histories.

What complexities arose when building a reusable prediction pipeline with customer-specific model training?

One customer’s trained model would not necessarily perform well for another, yet asking every customer to build a pipeline would defeat the purpose of the templates. The team needed a common structure for turning incident records into features, labels, predictions, and explanations while allowing the learned weights to differ by customer. The solution was to ship the training contract rather than fixed weights. The template carries Data Model Object (DMO) API names, transformation logic, and model structure, while each customer’s incident history determines the learned weights and parameters.

That contract begins in Data 360, where incidents, milestones, tasks, knowledge articles, and configuration items sync into DMOs. A daily data transform flattens and labels those records into one training object. For SLA Breach Prediction, it converts milestone violation data, IsViolated, into a binary 0/1 label. Einstein on Data 360 then trains an XGBoost binary classification model on that object. A scoring data transform writes predictions and the top three predictors into a score DMO, which the AI Accelerator card displays on the incident page. Models train in the home org, while companion orgs consume the results.

By normalizing customer records into the expected shape, Data 360 connects data preparation, model training, inference, and presentation through the same template. Customers can use the workflow without rebuilding the pipeline or receiving another customer’s trained model.

What prevented immediate incident-risk predictions, and how did customer data quality affect reliability?

Immediate predictions faced two constraints: scoring latency and customer data quality. A new incident could wait up to an hour for its score, while thin or poorly maintained histories could produce less trustworthy predictions. A reusable template could not eliminate either limitation.

For scoring, the team shipped a scheduled batch approximately once an hour, meeting the original product requirements. The real-time approaches available during development raised scalability concerns for large customers, including potential job failure. Batch processing delivered scoring within those constraints, while immediate predictions remained an anticipated customer need. For training, the team required approximately 400 completed incidents and documented the customer-dependent data limitations. Each model learns from its customer’s Data 360 history, so the quality of that history remains essential to the quality of its predictions.

The templates went generally available in February. At the time of this interview, the team had not received reports of these anticipated challenges becoming production problems and was still awaiting more customer feedback. Development evidence remained scenario-based.

The approach assigned a distinct role to each part of the system: synthetic scenarios tested predictive behavior, the template defined the training and scoring workflow, and customer history supplied the learned parameters. Those boundaries let the team develop predictive AI models before customer data was available while keeping each deployed model’s learning specific to the customer.

Learn More

Related Articles

View all