Changing the connection string during the TFS build for the Publish Profile of the SSDT (Sql Server Data Tool)


One of our customers needed to change that, they need to type the password for the SQL login inside the publish profile but of course they couldn’t do that in terms of security reasons, so we created that during the TFS  build as the following:

  • Create a PowerShell script that replace the connection string
  • Check-In the script into source control
  • Execute the PowerShell script during the build and before compiling the code

Create a PowerShell script that replace the connection string

The idea is to create a PowerShell script to search for the publish profile in the build directory on the build machine then, replace connection string with the one with the password, remember that we need to do that before compiling the SSDT so during running the SSDT deployment, it will use the modified file.

During the build I will send $pathToSearch which will be the build directory,$publishProfileString which will be set as argument in the build definition and so $sqlUserName and $sqlPassword.

Param(
 [string]$pathToSearch,
 [string]$publishProfileString,
 [string]$sqlUserName,
 [string]$sqlPassword

)

try
{
 #$pathToSearch="C:\Radwan"
 #$publishProfileString = "/p:SqlPublishProfilePath=../DAI.Infrastructure.DataAccess.Ef.Db\PublishProfiles/DAI.Infrastructure.DataAccess.Ef.Db.preprod.publish.xml"
 #$sqlUserName="vvvvvvvv"
 #$sqlPassword="00000000"

 $publishPostion = $publishProfileString.LastIndexOf("publish.");
 $publishStringWihoutPublish = $publishProfileString.Substring(0, $publishPostion - 1);
 $lastDotPostion = $publishStringWihoutPublish.LastIndexOf(".");
 $publishProfileStageName = $publishStringWihoutPublish.Substring($lastDotPostion+1, $publishStringWihoutPublish.Length - $lastDotPostion-1); 

 $searchFilter = "*." + $publishProfileStageName + ".publish.*"
 $replaceValue="User ID=" + $sqlUserName +";password="+$sqlPassword+";"

 gci -Path $pathToSearch -Filter $searchFilter -Recurse | %{
 Write-Host " -> Changing $($_.FullName)"

 # remove the read-only bit on the file
 sp $_.FullName IsReadOnly $false

 # read the file content
 $results = Get-Content $_.PSPath
 Write-Host " -> File content: $($results)"

 # run the regex replace
 #(gc $_.FullName) | % { $_ -replace 'ID=user;', "User ID=""$sqlUserName"";password=""$sqlPassword"";" }| sc $_.FullName
 (gc $_.FullName) | % { $_ -replace 'User ID=user;', $replaceValue }| sc $_.FullName

 Write-Host "Done!"

 }
}
catch {
 Write-Host $_
 exit 1
}

Check-In the script into source control

So the script can be downloaded during the build as I will need to run it from the build machine

Execute the PowerShell script during the build and before compiling the code

SSDT change connection string

Migration to VSO (Visual Studio Online) with different tools


I had an assignment which required migrating all TFS on-premises to VSO (Visual Studio Online), this includes TSF2010, TFS2012 and TFS2013

I will try to explain most of  the process and the steps.

  1. Problem
  2. Migrating work-items using OpsHub
    1. Solution Summary
    2. Solution Details
    3. Solution Notes
  3. Migrating work-items using TFS Integration Platform
    1. Solution Summary
    2. Solution Details
    3. Solution Notes
  4. Migrating Source control using OpsHub
    1. Solution Summary
    2. Solution Details
    3. Solution Notes
  5. Migrating Source control using TFS Integration Platform
    1. Solution Summary
    2. Solution Details
    3. Solution Notes
  6. Overall Notes
  7. Links

1.Problem

VSO (Visual Studio Online) doesn’t support customization yet.
The first server for my assigment was TFS-2010 and most projects were customized based on combination between MSF for CMMI process template v.4.0, v.5.0 and also some work-items from different Agile templates

2.Migrating work-items using OpsHub

1.Solution Summary

The free version of OpsHub required the process template of the project matching an existing one of the standard process templates (Agile, CMMI or Scrum), so in order to migrate a project we need to transform it into Agile v.5.0 (TFS2010) and migrate work-items as the following:

  • Importing work-items that doesn’t exist (Issue) from Agile v.5.0
  • Copy existing work-items that not belong to Agile into another work-items types of Agile v.5.0
    • (Change request –> Issue)
    • (Requirement –> User Story
  • Change Categories by removing the usage of none Agile work-items
  • Destroy none Agile work-items types (Change request – Requirement)
  • Change some of the system fields name to match the new work-items types of the Agile v.5.0
    Import all Agile v.5.0 work-items (User Story, Bug, Test Case, Issue, Shared Steps)
  • Verify that current project matching 100% of the Agile v.5.0
  • Running work-items migration using OpsHub tool

2.Solution Details

  • Importing work-items that doesn’t exist (Issue) from Agile v.5.0

Downloaded Agile v.5.0 process template from TFS
image001

image002

Open TFS Team Project Manager, select the project and navigate to Work item configuration/work items types/import

Click add and select Issue work-item from Agile v5.0 work-items and click on import
image003
I could also import it using WitAdmin command line
image004

  • Copy existing work-items that not belong to Agile into another work-item types of Agile v.5.0
    • (Change request –> Issue)
    • (Requirement –> User Story
  1. Make a query includes all the work-items we want to copy into other types
  2. Open the query result in Excel
  3. Copy all rows except the Work Item Type
  4. Change the Work Item Type into the desired type
  5. Click publish to publish new work-items

image005

  • Change Categories by removing the usage of none Agile work-items
  1. The Requirement category was reference Requirement work-item as the default work-item and since we will remove that work-items as it’s not belong to Agile v5.0 and also to make the categories match Agile v5.0 process template we had to change the category to reference User Story instead of Recruitment
  2. Open TFS Team Project Manager, select the project and navigate to Work item configuration/Categories/Update & import/Load From Team Project and select the current project
  3. Change the default work-item for Requirement category from Requirement into User Story
  4. Click Import work-items category

image006

  • Destroy none Agile work-items types (Change request – Requirement)

This also will delete all work-items from TFS DB for those types.

Run the following 2 commands from WitAdmin command line tool

witadmin.exe destroywitd /collection:http://kbndvtfs01:8080/tfs/TempVSO2 /p:ServicesVSO /n:”Requirement”
witadmin.exe destroywitd /collection:http://kbndvtfs01:8080/tfs/TempVSO2 /p:ServicesVSO /n:”Change Request”

image007

  • Change some of the system fields name to match the new work-items types of the Agile v.5.0

Test Case:
**********
witadmin changefield /collection:http://kbndvtfs01:8080/tfs/TempVSO2 /n:”System.ExternalLinkCount” /name:”External Link Count”
witadmin changefield /collection:http://kbndvtfs01:8080/tfs/TempVSO2 /n:”System.RelatedLinkCount” /name:”Related Link Count”
witadmin changefield /collection:http://kbndvtfs01:8080/tfs/TempVSO2 /n:”System.HyperLinkCount” /name:”Hyperlink Count”
witadmin changefield /collection:http://kbndvtfs01:8080/tfs/TempVSO2 /n:”System.AttachedFileCount” /name:”Attached File Count”
witadmin changefield /collection:http://kbndvtfs01:8080/tfs/TempVSO2 /n:”System.AreaId” /name:”Area ID”

Shared Steps:
*************
witadmin changefield /collection:http://kbndvtfs01:8080/tfs/TempVSO2 /n:”System.IterationId” /name:”Iteration ID”

image008

  • Import all Agile v.5.0 work-items (User Story, Bug, Test Case, Issue, Shared Steps)
  1. Open TFS Team Project Manager, select the project and navigate to Work item configuration/work items types/import
  2. Click add and select all Agile v5.0 work-items and click on import

image009

  • Verify that current project matching 100% of the Agile v.5.0
  1. Open TFS Team Project Manager, select the project and navigate to Work item configuration/compare
  2. Click add and select all Agile v5.0 work-items and click on Compare Team Project With Source
  3. Make sure it 100%
  • Running work-items migration using OpsHub tool

Make sure that the account that you logged-in and will be used for the migration added in Project Collection Service Accounts group.image011

Make sure that you have a project with the same name and with the process template that match the desired process templateimage012

Run OpsHub and select New Migration
image013
Select the source collection (local) and the destination account (VSO)
image014
Choose I want to migrate work-item and click on Next
image015

Select the destination project on VSO (Visual Studio Online) and click Next
image016
Map the local users to the online users and click Next
image017
image018

Make sure that the verification running successfully
image019
Start the migration
image020
Review that the migration ran successfully
image021

3.Solution Notes:

  • OpsHub free version doesn’t support migration from TFS on- premises to TFS on-premises
  • OpsHub free version required that you migrate to the same project name
  • Migrating (copying work-items) using MS Excel will lose the history and the attachments

 

3.Migrating work-items using TFS Integration Platform

1.Solution Summary

Using TFS Integration Platform we can make custom mapping to map work-items to different types and also mapping work-item’s fields to different fields, so we will not changing the process template of the project also I will not map all fields so I can show how to resolve conflicts for undefined fields

  • Run TFS Integration Platform
  • Use a custom mapping file (Doesn’t map all required fields)
  • Show how to resolve conflict for none found fields
  • Continue migration

2.Solution Details

  • Run TFS Integration Platform

After launching the tool, click on Create New
image022
Select workItemTracking.xml
image023
In the left side select the source (on-prem), in the right side select VSO (Visual Studio Online), make sure that workflow type is One-way-migration, click on Edit XMLimage024

Put the mapping file
image025

Review how the mapping file translated by TFS Integration Tool, click on each work-item mapping to see the details of the mapping fields, click save to DBimage026

Click on Start
image027

The first conflict raised because a field in a work-item from the source not exist in the destination work-item, we can choose the first option which include typing the field that we want to map source field and click resolve.
image028
I chose second option to ignore that filed

image029
Make sure that the conflict resolved (green icon) and click start
image030
The second conflict raised because the second field that not exist, I will solve this conflict with another way, click on View Rules
image031
Double click on the rule to view the rule details, review the information, take care of the scope that identify where to apply the rule
image032
To solve the conflict, right click on the rule and select copy rule.
image033

In the InvalidFieldRefreneceName I will put the name of the missing field, I also put it as the description but this is not important, I will leave the scope with no change as I only need that scope.
image034
Click preview, this will display all work-items under that scope, select the work-item and click resolve
image035
Review that second rule created successfully after resolve the conflict, click start to continue the migration process
image036

3.Solution Notes:

  • When you resolve a conflict, this built a rule, you may want to create a rule to solve conflict in case you want to change the default scope for the automatically created rule.
  • You don’t need to restart the migration as the tool can continue for the conflicted items
  • Be careful of how to exclude fields in the mapping file
  • TFS Integration platform not support new features anymore

 

3.Migrating Source control using OpsHub

1.Solution Summary

Just run the free version of OpsHub to migrate the source code

2.Solution Details

Make sure that the account that you logged-in and will be used for the migration added in Project Collection Service Accounts group.
image011
Make sure that you have a project with the same name, it’s not important to match the process template unless you plan to migrate work-items using the free version of OpsHub too
image012
Run OpsHub and select New Migration
image013
Select the source collection (local) and the destination account (VSO)
image014

Choose I want to migrate version control data and click on Next
image037

Select the destination project on VSO (Visual Studio Online) and click Next
image016
Map the local users to the online users and click Next
image017
image018

Make sure that the verification running successfully
Start the migration

Review that the migration ran successfully
image038

3.Solution Notes:

  • You must add all users with their license before migration take place so we can map them to the local users as the tool only shows the licensed users
  • The tool heavily consumed CPU hence it needs a good machine with good CPU and RAM, it took about 3 hours to migrate source code that took only 10 minutes to be migrated using TFS Integration Platform

4.Migrating Source control using TFS Integration Platform

1.Solution Summary

Just run the TFS Integration platform and migrate the code

2.Solution Details

Make sure that the account that you logged-in and will be used for the migration added in Project Collection Service Accounts group.
image011

  • Run TFS Integration Platform

After launching the tool, click on Create New
image022
Select VersionControl.xml
image039

In the left side select the source (on-prem), in the right side select VSO (Visual Studio Online), make sure that workflow type is One-way-migration
image040

Because when we create a new project on VSO it create the folder “BuildProcessTemplate”, this makes a conflict during the migration as it exists on both sides(Source and Destination) the manual resolve couldn’t solve this conflict.
image041

We could destroy those folders on any side to solve the conflict but I thought it’s better to just map all other folders except that one
image042

Start the migration
image043

If your source code use branches you may encounter the following conflict “VC path not mapped conflict type” and because we want the history as much as possible, we will chose “Add’ for ‘Branch’, by skipping for ‘Merge’ and by changing to ‘Add’ for Remove” and click resolve.
Remember that we could change the resolving scope by create a rule instead of resolving by applying the automatic rule
image044

Continue the migration by clicking on Start
image045

 

6.Overall Notes:

WARNING: Always do the migration on a test environment first before you ever touch production and before touch the production always backup first.
It’s recommended that developers test the migration on a testing environment before making the live migration
VSO (Visual Studio Online) doesn’t support multiple collection yet but we can create multiple accounts that has different default collection

7.Links:

Migrating On Premises TFS to VS Online (Brian Harry)
Migrating Your Data from TFS to Visual Studio Online with New Free Utility from OpsHub (Ed Blankenship)
Migrate team projects from on-premises TFS to Visual Studio Online (MSDN)
TOC: TFS Integration Tools Blog Posts and Reference Sites
TFS Integration Platform – Summary of Links
How to Assign licenses to users on VSO
How to integrate VSO with Azure AD

Create a TFS Build Custom Activity to Read From File


I needed to create a Custom Activity that read from XML file and display the values on the Build log.

It’s very easy and I will not re-write how to do the basic Custom Activity, but if you are not familiar with that, you can see the following link:
Use and develop custom build process activities

Here, I needed to add OutArgument to return the value that I got from reading the file, I will got the value from the file and assign that value to OutArgument.
OutArguemnt in TFS Custom activity

After that, I built the project and distribute the library (DLL) under source control where the build controller point for the custom assemblies, see the previous link for the MSDN on how to do that.

Open your process template that you want to use the new custom activity and Right-click on the Toolbox and browse to the created library as the following image.
Add custom activity for The Visual Studio Toolbox

Drag and drop your new custom activity, declare a new variable (MyValue) to hold the return value from the reading process as the following image.
ReadFromXML Activity

The Build summary will be as the following image.
Build Summary

The build log will be as the following image.
Build Log

 

Make the activity return list of string

We will need to define OutArgument as List<string> and assign that in the execute method
OutArgument as list of string

Declare a variable (MyValue) but this time the type is List<string> too
Drag and drop For each Activity of string and print each item in the list as the followingReadFromXML Activity for list of string

 

The Build summary will be as the following image.
Build Summary for list of string

Versioning Assembly during TFS Build 2013


My fellows MVPs Colin Dembovsky and Ricci Gian Maria had great posts on doing Assembly Versioning, it doesn’t cover my case but they helped me very well. Thanks guys :-)

First, It’s better to understanding the Assembly version.

Assembly versions consist of four different parts ({Major Version}.{Minor Version}.{Build Number}.{Revision Source Control Revision})

Major.Minor.Build.Revision

Major: The main version of the product. For example the Major of all the VisualStudio 2012 assemblies are 11 and VisualStudio 2013 are 12.

Minor: Small changes that doesn’t required changing in Application interfaces, so it will not break other application it depends on. This number will be reset to zero when the Major number changes.

Build: Using this number you can find the build on the build server.

Revision: This is the number from source control for that build number.

See the following image:
VS Version

We can see in the previous image that the Version is different from the File Version? so why this happen?

To answer this question we need to know that you don’t have to do that, but as the Version is the version that .NET uses when linking assemblies, so you might want to keep the Version without the build number and the revision as you may want to change something, it might be only changing in the build parameters and you don’t want to rebuild all dependent assemblies.


Using one AssemblyInfo.cs for all projects within the same solution

We might want to use one file to maintain the Version and the File version for all projects as the following:

  • Create AssemblyInfo.cs as a solution file, it will include AssemblyVersion and AssemblyFileVersion attributes
  • Add an exiting item for all your projects as a linked item to point to that file
  • Remove the AssemblyVersion and AssemblyFileVersion attributes from all the project AssemblyInfo.cs files.

Add exiting item as link

 

Versioning Assembly during TFS Build

The idea hear is to change the number in the AssemblyInfo.cs during the build and before the compiling happen.

In my case I use a PowerShell script and I customize my process as the following:

Param(

[string]$pathToSearch,
[string]$buildNumIncludeBuildDefination,
[string]$extractedBuildNum,
[string]$chagesetWithC,
[string]$chageset,
[string]$searchFilter = "AssemblyInfo*.*",
[string]$assemblyVersion,
[string]$fileAssemblyVersion

)

try
{

$extractedBuildNum = $buildNumIncludeBuildDefination.substring($buildNumIncludeBuildDefination.LastIndexOf(".")+1 ,(($buildNumIncludeBuildDefination.length)- ($buildNumIncludeBuildDefination.LastIndexOf("."))-1))
$chageset = $chagesetWithC.substring($chagesetWithC.LastIndexOf("C")+1 ,(($chagesetWithC.length)- ($chagesetWithC.LastIndexOf("C"))-1))

$assemblyVersion = $assemblyVersion.Replace("B", $extractedBuildNum).Replace("R", $chageset)
$fileAssemblyVersion = $fileAssemblyVersion.Replace("B", $extractedBuildNum).Replace("R", $chageset)

gci -Path $pathToSearch -Filter $searchFilter -Recurse | %{
Write-Host " -> Changing $($_.FullName)"

# remove the read-only bit on the file
sp $_.FullName IsReadOnly $false

# run the regex replace
(gc $_.FullName) | % { $_ -replace 'AssemblyVersion\("[0-9]+(\.([0-9]+|\*)){1,3}"\)', "AssemblyVersion(""$assemblyVersion"")" }| sc $_.FullName
(gc $_.FullName) | % { $_ -replace 'AssemblyFileVersion\("[0-9]+(\.([0-9]+|\*)){1,3}"\)', "AssemblyFileVersion(""$fileAssemblyVersion"")" }| sc $_.FullName
Write-Host "Done!"
Write-Host $assemblyVersion
Write-Host $fileAssemblyVersion
Write-Host $extractedBuildNum

}
}
catch {
Write-Host $_
exit 1
}

I opened the process template and within “Try Compile and Test” I added If sequence activity 
In Try and Compile I add if squence

The main activity inside the “If” activity will be InvokeProcess as the following:
VersionAssembly InvokeProcess

I sent argument to the PowerShell script as the following:
Argument for PowerShell

To set parameters in the build definition, see the following image:
Dev_Build_Version

Remember if you want to not change the Version number, remove B.R and Add 0.0 instead ex (2.5.0.0)

Download the Process Template

Download PowerShell Script

Creating and Deploying Web Package during TFS Build 2013


I have an old post that do the same concept with TFS 2010 but as usual there are some differences with TFS 2013:-)

How to run remote deploy with MS Deploy

I tried many ways from many blog posts, they are almost the same but none of them worked for me, so I decide to create my own solution and share that include the Build Process Template
Of course I needed to install MSDeploy (Web Deploy 3.5) on the Build machine

Customize the process template as the following:

Note: I didn’t include the error handling during the customization, you may need to do that :-)

  • Adding ConverWorkspaceItem and If activities, we could add ConverWorkspaceItem part of the If sequence, this even could be better, you can do that too :-)
    ConvertWorkspaceItem and Create packge with Deploy
  • The ConverWorkspaceItem will transform the web application selected in the build definition to a local path
    ConverWorkspaceItem
  • In the If activity the main 2 activities are, Create Package (MSBuild activity) and Deploy Package (InvokeProcess activity)
    Create and Deploy package workflow
  • The Create Package activity will be as the following:
    • Configuration —> the configuration that entered in the build definition
    • CommandLineArgument —->  /t:build;publish
    • Project —-> the project that transformed from ConverWorkspaceItem activity

Create Package - MSBuild

  • The Deploy Web Package (InvokeProcess) activity will be as the following image:
    Deploy Web Package InvokeProcess
  • The created package on the working directory on the build agent will be as the following:
    Web Package
  • The Deployment section on the build definition will be as the following:
    Dev_Build_Deploy
  • How to test the create deployment file on the build agent
    Run file.Deploy.cmd
  • How to publish without a package using a publish profile
    Run MSBuild with publish profile
  • How to add a parameters value inside the parameter file.xml that created with the package
    Create Web package and add one paramter in the file

Download the process Template

 

Debugging TFS Job Agent Extension


Creating TFS Job Agent was not difficult at all but debugging it was a pain in the nick! :-)

here I explain how to find what the error and how to fix that

Extension not found for TFSJobAgent (ExtensionNotFound)

So first how to know if my job agent succeeded or failed and if it’s failed, why this happen?

How to see this

Open the following link:

http://TFSServer:8080/tfs/_oi/

Click on the job history, if the job failed you will see it here and you can click on the small arrow to see History or DefinitionTFS Job Definition and History

History
TFS Job History Details

Definition
TFS Job Definition Details

In case there is nothing in the history you should see your job on the Job Motoring page as the following:
TFS Job Monitoring

Remember you will need to restart the TFS Job Agent every time you deploy a new version of the dll so the service can load the assembly and this the reason for the fail first time.
Restart TFS Job Agent

Links:

How to see activity and job history in TFS 2012

TFS2012: New tools for TFS Administrators

 Identity Synchronization in Team Foundation Server 2010