Include referenced libraries as CLR assembly in Dacpac (afterwards)

[jvdslikke]

Hi, I was looking for a way to include another project with SQL CLR code as Assembly in the Dacpac. Because I didn't like to have it in Postdeployment scripts and also didn't want to keep an old SSDT project. I managed to get it working by adding support for DLL references.

I read all the information in microsoft/DacFx#523. To work around the "SQL70557: This assembly is corrupt or not valid" I add the assembly directly to the Dacpac ZIP afterwards and fix the references. Build errors "unresolved reference to Assembly" are ignored (because the assembly is added afterwards, it is missing during build).

The referenced project must have .NET Framework as TargetFramework. Because SQL Server only supports .NET Framework assemblies. (This is not validated during build.)

This draft PR is a working proof-of-concept. Are you willing to accept this functionality? Then I can make it into a full PR by improving the code, adding tests, etc.

[jmezach]

@jvdslikke First of all, thanks for taking a stab at this. While we don't use SQLCLR ourselves I do realise there are many folks out there that do, so having a solution for this in the box would definitely be nice. But to be honest, looking at this it feels a bit hacky and I'm wondering if it is worth maintaining in its current form. We are making a lot of assumptions here about the inner workings of DacFx that could be changed at any time and then this would break.

Now we are already making some assumptions by calling internal API's from DacFx through reflection and that got me thinking if there might actually be some internal API already that adds a SQLCLR assembly to the model. Apparently sqlpackage is capable of deploying one, so I wouldn't be surprised if DacFx has a method somewhere to add it to the model. I would much rather call an internal API that does the heavy-lifiting than trying to do the heavy-lifting ourselves. That might also mean we don't have to work around the warnings as well.

Aside from that I do think we should treat these references as a separate thing, rather than trying to shoehorn them in the existing references and then checking whether it is a .dll or a .dacpac. That means adding a new command line option to the DacpacTool and modifying the Sdk.targets to pass these along and essentially creating a new <ItemGroup> in the project file. Perhaps we can leverage <Reference> for that, but I'm not sure.

[jvdslikke]

Hi @jmezach thanks for looking into my PR.

I agree that the current solution is hacky. I tried adding the assembly with passing a CREATE ASSEMBLY script to TSqlModel.AddOrUpdateObjects. Unfortunately this throws an exception "SQL70557: This assembly is corrupt or not valid". From microsoft/DacFx#523 I understand this is because it validates the assembly with the framework executing the build, which is .NET Core, while a CLR Assembly must be .NET Framework.

Although the solution is hacky, it only impacts those actually using SQL CLR.

Another option might be to rewrite the DacpacTool to .netstandard/.NET Framework so the build can be executed with .NET Framework. But I assume that is not something you want to consider 😊.

I am willing to explore the option of calling an internal DacFX API. It there documentation somewhere about what internal methods DacFX offers which can be called through reflection?

Aside from that I do think we should treat these references as a separate thing, rather than trying to shoehorn them in the existing references and then checking whether it is a .dll or a .dacpac. That means adding a new command line option to the DacpacTool and modifying the Sdk.targets to pass these along and essentially creating a new <ItemGroup> in the project file. Perhaps we can leverage <Reference> for that, but I'm not sure.

I do not have much knowledge about MSBuild, but I will give it a try.

[jmezach]

So what you're saying is that even though you have a .NET Framework assembly and you put that into a CREATE ASSEMBLY script as a binary literal and then pass that to AddOrUpdateObjects the validation fails if DacFx is running in a .NET Core host, but succeeds if it runs in a .NET Framework host? That is interesting.

If that is the case I'm wondering if we could add a new .NET Framework CLI tool that would add the assemblies to an existing .dacpac that was created by the DacpacTool previously. That tool would obviously only work when running on Windows, but at least we'd have a lot less code to maintain.

[jvdslikke]

So what you're saying is that even though you have a .NET Framework assembly and you put that into a CREATE ASSEMBLY script as a binary literal and then pass that to AddOrUpdateObjects the validation fails if DacFx is running in a .NET Core host, but succeeds if it runs in a .NET Framework host? That is interesting.

Yes that is what I understood from microsoft/DacFx#523, and have now proven with a small test project (attached)😀.

The command dotnet run --project .\DacpacWithAssemblyCreationTest\DacpacWithAssemblyCreationTest.csproj fails with the error "This assembly is corrupt or not valid".

When you change the TargetFramework in DacpacWithAssemblyCreationTest.csproj to "net48" the command above succesfully produces a Dacpac with the assembly included.

If that is the case I'm wondering if we could add a new .NET Framework CLI tool that would add the assemblies to an existing .dacpac that was created by the DacpacTool previously. That tool would obviously only work when running on Windows, but at least we'd have a lot less code to maintain.

That is a wonderful idea, better than hacking into the Dacpac ZIP or rewriting the whole DacpacTool to .NET Framework. Then still we need to ignore "unresolved reference to Assembly" in the Dacpactool because the assembly is added afterwards.

I will update my PR to this idea.

[jmezach]

@jvdslikke What if we'd run the .NET Framework tool first to create a .dacpac with just the assembly in it, then we let the existing .NET Core DacpacTool add the rest to it? Then I would hope that we don't get the warning, since the assembly should already be there.

[jvdslikke]

@jmezach I think it will give the same error when the DacpacTool opens and saves the .dacpac, because it validates the whole model including the assembly. But I will give it a try.

[jvdslikke]

I vibe-coded a solution with a "DacpacToolFramework" which first makes a Dacpac with the assemblies and then passes it to the DacpacTool. But this gives the "This assembly is corrupt or not valid" in the validation as expected because it validates with the .NET Core tool.

I kept a backup in a branch (https://github.com/MontaServices/MSBuild.Sdk.SqlProj/tree/feature/assembly-ref-framework-before) and will work further on a solution to add the assembly afterwards to the Dacpac.

[jvdslikke]

I implemented (vibe-coded) the solution to add the assembly afterwards to the dacpac with the DacpacToolFramework. It also adds objects referencing the assembly afterwards. This produces a dacpac without errors. But I observed that the relationship from objects referencing the afterwards-added objects are lost in the Dacpac. See comment #915 (comment) for details.

I will work further on finding a solution for this. Maybe we should also re-add all objects referencing objects referencing the assembly.

[jmezach]

I vibe-coded a solution with a "DacpacToolFramework" which first makes a Dacpac with the assemblies and then passes it to the DacpacTool. But this gives the "This assembly is corrupt or not valid" in the validation as expected because it validates with the .NET Core tool.

I kept a backup in a branch (https://github.com/MontaServices/MSBuild.Sdk.SqlProj/tree/feature/assembly-ref-framework-before) and will work further on a solution to add the assembly afterwards to the Dacpac.

Just out of curiosity, at what point does the DacpacTool throw the "This assembly is corrupt or not valid" error? Is that when we add additional objects to the model? Or does that happen at a later step? Perhaps you can share a stack trace for that? I still think that adding the assembly first and then adding the rest would be the nicest approach to this

[jvdslikke]

@jmezach I can't produce a stack trace at the moment. The error throws first at PackageBuilder.ValidateModel. If it will be ignored there, it throws at PackageBuilder.SaveToDisk -> DacPackageExtensions.BuildPackage. That method has the option to ignore validation errors (with PackageOptions.IgnoreValidationErrors). The error codes are documented here. I tried "SR0025" but that doesn't work to bypass the validation error.

This PR now has the solution to add the assembly and all objects that reference it afterwards with a DacpacToolFramework. It also includes a test and passes the files with a separate build target. Are you willing to accept this?

[jmezach]

@jvdslikke What if you pass SQL70557 to IgnoreValidationErrors?

[jvdslikke]

@jmezach I created a version to test your suggestion but it doesn't work. Error with stack trace:

Unhandled exception. Microsoft.SqlServer.Dac.DacServicesException: Cannot save package to file. The model has build blocking errors:
EXEC : error SQL70557: Error validating element [SqlClrTestLibrary]: This assembly is corrupt or not valid. [C:\Users\Johan.vanderSlikke\source\repos\MSBuild.Sdk.SqlProj\test\TestProjectWithSqlClrReference\TestProjectWithSqlClrReference.csproj]
  
     at Microsoft.SqlServer.Dac.Model.SqlSchemaModelObjectService.ValidateAndThrowOnErrors(Boolean treatWarningsAsErrors, IEnumerable`1 ignoreValidationErrors)
     at Microsoft.SqlServer.Dac.Model.SqlSchemaModelObjectService.BuildPackage(Func`1 streamGetter, PackageMetadata packageMetadata, PackageOptions packageOptions)
     at Microsoft.SqlServer.Dac.Model.TSqlModel.BuildPackage(String packageFilePath, PackageMetadata packageMetadata, PackageOptions packageOptions)
     at Microsoft.SqlServer.Dac.DacPackageExtensions.BuildPackage(String packageFilePath, TSqlModel model, PackageMetadata packageMetadata, PackageOptions packageOptions)
     at MSBuild.Sdk.SqlProj.DacpacTool.PackageBuilder.SaveToDisk(FileInfo outputFile, PackageOptions packageOptions) in C:\Users\Johan.vanderSlikke\source\repos\MSBuild.Sdk.SqlProj\src\DacpacTool\PackageBuilder.cs:line 234
     at MSBuild.Sdk.SqlProj.DacpacTool.Program.BuildDacpac(BuildOptions options) in C:\Users\Johan.vanderSlikke\source\repos\MSBuild.Sdk.SqlProj\src\DacpacTool\Program.cs:line 154
     at MSBuild.Sdk.SqlProj.DacpacTool.BuildOptions.RunAsync() in C:\Users\Johan.vanderSlikke\source\repos\MSBuild.Sdk.SqlProj\src\DacpacTool\BuildOptions.cs:line 84
     at MSBuild.Sdk.SqlProj.DacpacTool.GeneratedCode.BuildOptionsBuilder.<>c__DisplayClass1_0.<<DoBuild>b__2>d.MoveNext() in C:\Users\Johan.vanderSlikke\source\repos\MSBuild.Sdk.SqlProj\src\DacpacTool\obj\Debug\net10.0\DotMake.CommandLine.SourceGeneration\DotMake.CommandLine.SourceGeneration.CliIncrementalGenerator\BuildOptionsBuilder-cae2z5g.g.cs:line 484
  --- End of stack trace from previous location ---
     at System.CommandLine.Invocation.InvocationPipeline.InvokeAsync(ParseResult parseResult, CancellationToken cancellationToken)
     at DotMake.CommandLine.CliParser.RunAsync(String[] args, CancellationToken cancellationToken) in D:\Development\Projects\DotMake-Build\command-line\src\DotMake.CommandLine\CliParser.cs:line 201
     at DotMake.CommandLine.Cli.RunAsync[TDefinition](String[] args, CliSettings settings, CancellationToken cancellationToken) in D:\Development\Projects\DotMake-Build\command-line\src\DotMake.CommandLine\Cli.cs:line 133
     at MSBuild.Sdk.SqlProj.DacpacTool.Program.Main(String[] args) in C:\Users\Johan.vanderSlikke\source\repos\MSBuild.Sdk.SqlProj\src\DacpacTool\Program.cs:line 24
     at MSBuild.Sdk.SqlProj.DacpacTool.Program.<Main>(String[] args) 

[jvdslikke]

@jmezach I updated the PR with:

• Docs
• An option to add SQL statements to the predeploy script to trust the assembly.

[jvdslikke]

@jmezach Actually I don't really like this solution to the problem myself, it still feels like a hacky workaround.

I created another solution which does the whole Dacpac creation in .NET Framework when assemblies are referenced. See #924.