String is not supported by database collation - Emoji in name

[sajawalnauman]

Hello, I am getting this error when running the tool.

!! Check for latest version - We fix issues constantly - If not, please create a discussion on https://github.com/nkdAgility/azure-devops-migration-tools/discussions so we can get this fixed !!
---> (Inner Exception #0) Microsoft.TeamFoundation.WorkItemTracking.Client.UnexpectedErrorException: There is a problem on the server. Contact your Azure DevOps Server administrator. ---> System.Web.Services.Protocols.SoapException: %error="600177";%:AddConstant: The following string is not supported by database collation: %String="Mike Metheny ??";%.
at Microsoft.TeamFoundation.WorkItemTracking.Proxy.RetryHandler.HandleSoapException(SoapException se)
at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)
--- End of inner exception stack trace ---
at MigrationTools.Processors.TfsWorkItemMigrationProcessor.d__31.MoveNext() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 650<---
Error logged to Elmah.io!
!! Check for latest version - We fix issues constantly - If not, please create a discussion on https://github.com/nkdAgility/azure-devops-migration-tools/discussions so we can get this fixed !!
--- End of inner exception stack trace ---
at MigrationTools.Processors.TfsWorkItemMigrationProcessor.d__31.MoveNext() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 650<---
Error logged to Elmah.io!
!! Check for latest version - We fix issues constantly - If not, please create a discussion on https://github.com/nkdAgility/azure-devops-migration-tools/discussions so we can get this fixed !!
[11:45:42 INF] [16.3.3] TfsWorkItemMigrationProcessor completed in 00:04:14.2426260

The name in Source project is Mike Metheny 🐓. The tool and config file works fine for work items that dont have the name Mike Metheny 🐓in them and migrates them properly, but for work items that do have this name as a field or as a text anywhere in the ticket, I am getting this error. I have tried the String Manipulator Tool multiple times according to the documentation but I am still getting the same error, and the manipulator tool is not replacing the string. Please assist.

"StringManipulatorTool": {
"Enabled": true,
"Manipulators": [
{
"$type": "RegexStringManipulator",
"Description": "Remove emojis but preserve Unicode letters and symbols",
"Enabled": true,
"Pattern": "[\uD800-\uDBFF][\uDC00-\uDFFF]|\uFE0F",
"Replacement": " "
}
]
}

[MrHinsh]

An easy way is to get the user to remove the emoji!

Have you tested your regex against the text?

[MrHinsh]

You couldtry:

"StringManipulatorTool": {
  "Enabled": true,
  "Manipulators": [
    {
      "$type": "RegexStringManipulator",
      "Description": "Remove non-BMP characters (including emojis)",
      "Enabled": true,
      "Pattern": "[^\\u0000-\\uFFFF]",
      "Replacement": ""
    }
  ]
}

[sajawalnauman]

@MrHinsh These are users that used to work in the org a few years ago, but do not anymore, but their names still exist in the ticket fields and history. There are multiple of them, along with Mike Metheny like

I tried running it with the code you sent, but it is still giving the error. I want to remove the emojis any and every time they show up in the ticket.

[10:16:47 ERR] [16.3.3] Microsoft.TeamFoundation.WorkItemTracking.Client.UnexpectedErrorException: There is a problem on the server. Contact your Azure DevOps Server administrator. ---> System.Web.Services.Protocols.SoapException: %error="600177";%:AddConstant: The following string is not supported by database collation: %String="Mike Metheny ??";%.
at Microsoft.TeamFoundation.WorkItemTracking.Proxy.RetryHandler.HandleSoapException(SoapException se)
at Microsoft.TeamFoundation.WorkItemTracking.Proxy.WorkItemServer.Update(String requestId, XmlElement package, XmlElement& result, MetadataTableHaveEntry[] metadataHave, String& dbStamp, IMetadataRowSets& metadata)
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)
--- End of inner exception stack trace ---
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SendUpdatePackage(XmlElement package, XmlElement& result, Boolean bulk)
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.SaveWorkItems(WorkItem[] workitems, Boolean bulk, SaveFlags saveFlags)
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.Save(SaveFlags saveFlags)
at MigrationTools.TfsExtensions.SaveToAzureDevOps(WorkItemData context) in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\TfsExtensions.cs:line 83
at MigrationTools.Processors.TfsWorkItemMigrationProcessor.ReplayRevisions(List`1 revisionsToMigrate, WorkItemData sourceWorkItem, WorkItemData targetWorkItem) in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs:line 846
[10:16:47 INF] [16.3.3] ===============================================================

[sajawalnauman]

I have tried many variations of the regex which filter out the emojis in a regex editor, those mentioned in the String Manipulator tool documentation and otherwise, but I still get the same error when the tool is run. This is the same for when "Mike Metheny 🐓" is mentioned, or any other user with an emoji in the name. There are dozens of users that have these emojis in source org, and hundreds if not thousands of work items which would include these names, either as a field or in the history somewhere, so it not possible to filter and exclude these work items completely.