2. Home
  3. Articles
  4. CRM
  5. Zoho
  6. Zoho Deluge: Handle Commas between Quotes in a CSV (and New Lines)

For Zoho Services only:


I'm actually part of something bigger at Ascent Business Solutions recognized as the top Zoho Premium Solutions Partner in the United Kingdom.

Ascent Business Solutions offer support for smaller technical fixes and projects for larger developments, such as migrating to a ZohoCRM.  A team rather than a one-man-band is always available to ensure seamless progress and address any concerns. You'll find our competitive support rates with flexible, no-expiration bundles at http://ascentbusiness.co.uk/zoho-support-2.  For larger projects, check our bespoke pricing structure and receive dedicated support from our hands-on project consultants and developers at http://ascentbusiness.co.uk/crm-solutions/zoho-crm-packages-prices.

The team I manage specializes in coding API integrations between Zoho and third-party finance/commerce suites such as Xero, Shopify, WooCommerce, and eBay; to name but a few.  Our passion lies in creating innovative solutions where others have fallen short as well as working with new businesses, new sectors, and new ideas.  Our success is measured by the growth and ROI we deliver for clients, such as transforming a garden shed hobby into a 250k monthly turnover operation or generating a +60% return in just three days after launch through online payments and a streamlined e-commerce solution, replacing a paper-based system.

If you're looking for a partner who can help you drive growth and success, we'd love to work with you.  You can reach out to us on 0121 392 8140 (UK) or info@ascentbusiness.co.uk.  You can also visit our website at http://ascentbusiness.co.uk.

Zoho Deluge: Handle Commas between Quotes in a CSV (and New Lines)

Zoho
Hits: 5184
What?
This is an article to demonstrate how to handle commas in a CSV file that were enclosed between two double-quotes. I've added to this the handling of new lines or carriage returns in between a pair of double-quotes.

Why?
Our use case is that we were trying to loop through rows of a CSV file which contained addresses which in turn contained commas. Saving this as a CSV and asking Deluge to parse the data in the appropriate columns was not working as expected.

How?
The quick answer is a regex that will replace any commas between two quotes with a custom string, to be exact:
copyraw 
v_FormattedData = r_Data.replaceAll("(\"[^\",]+)[,]([^\"]+\")","$1|mySpecialComma|$2",false);
  1.  v_FormattedData = r_Data.replaceAll("(\"[^\",]+)[,]([^\"]+\")","$1|mySpecialComma|$2",false); 

The slightly longer answer to describe this might be better explained if you consider the following snippet of code:
copyraw 
// sample data
v_Test = "00011,Joel Lipman,\"Flat 8, House Corner\",Brummieland";

// regex to replace a comma found between 2 double-quotes to a string of your choice
v_FormattedString = v_Test.replaceAll("(\"[^\",]+)[,]([^\"]+\")","$1|mySpecialComma|$2",false);

info v_FormattedString;
// yields: 00011,Joel Lipman,"Flat 8|mySpecialComma| House Corner",Brummieland

// split into a list (string delimited by commas)
l_StringParts = v_FormattedString.toList();

// show me column 3 with myspecialcomma replaced back to a comma
info l_StringParts.get(2).replaceAll("|mySpecialComma|", ",", true);

// yields: Flat 8, House Corner
  1.  // sample data 
  2.  v_Test = "00011,Joel Lipman,\"Flat 8, House Corner\",Brummieland"; 
  3.   
  4.  // regex to replace a comma found between 2 double-quotes to a string of your choice 
  5.  v_FormattedString = v_Test.replaceAll("(\"[^\",]+)[,]([^\"]+\")","$1|mySpecialComma|$2",false); 
  6.   
  7.  info v_FormattedString; 
  8.  // yields: 00011,Joel Lipman,"Flat 8|mySpecialComma| House Corner",Brummieland 
  9.   
  10.  // split into a list (string delimited by commas) 
  11.  l_StringParts = v_FormattedString.toList(); 
  12.   
  13.  // show me column 3 with myspecialcomma replaced back to a comma 
  14.  info l_StringParts.get(2).replaceAll("|mySpecialComma|", ",", true); 
  15.   
  16.  // yields: Flat 8, House Corner 

The long answer is to consider the following code which generates a sample CSV and then loops through storing each row as a data record and outputting it to screen:
copyraw 
// generate a sample CSV file
v_DataCSV = "1,Joel Lipman,Kings Castle,England\n";
v_DataCSV = v_DataCSV + "11,General Dogsbody,\"Flat 8, House Corner, My Street\",Brummieland\n";
v_DataCSV = v_DataCSV + "64,Elephant Man,\"123 New Street,\nNew York City\",USA\n";
f_CSVfile = v_DataCSV.toFile("test.csv");
//
// read and process CSV file
v_FileContent = f_CSVfile.getFileContent();
l_FileRows = List();
if(!isBlank(v_FileContent))
{
	l_FileRows = v_FileContent.toList("\n");
}
// loop through each row
for each r_Data in l_FileRows
{
	// initialize record
	m_Record = Map();
	m_Record.put("EmployeeID","");
	m_Record.put("Name","");
	m_Record.put("Address","");
	m_Record.put("Territory","");
	//
	// split values by commas
	l_FieldValues = r_Data.toList(",");
	if(l_FieldValues.size()>0)
	{
		m_Record.put("EmployeeID",l_FieldValues.get(0));
	}
	if(l_FieldValues.size()>1)
	{
		m_Record.put("Name",l_FieldValues.get(1));
	}
	if(l_FieldValues.size()>2)
	{
		m_Record.put("Address",l_FieldValues.get(2));
	}
	if(l_FieldValues.size()>3)
	{
		m_Record.put("Territory",l_FieldValues.get(3));
	}
	info m_Record.toString();
}
  1.  // generate a sample CSV file 
  2.  v_DataCSV = "1,Joel Lipman,Kings Castle,England\n"; 
  3.  v_DataCSV = v_DataCSV + "11,General Dogsbody,\"Flat 8, House Corner, My Street\",Brummieland\n"; 
  4.  v_DataCSV = v_DataCSV + "64,Elephant Man,\"123 New Street,\nNew York City\",USA\n"; 
  5.  f_CSVfile = v_DataCSV.toFile("test.csv"); 
  6.  // 
  7.  // read and process CSV file 
  8.  v_FileContent = f_CSVfile.getFileContent(); 
  9.  l_FileRows = List(); 
  10.  if(!isBlank(v_FileContent)) 
  11.  { 
  12.      l_FileRows = v_FileContent.toList("\n"); 
  13.  } 
  14.  // loop through each row 
  15.  for each r_Data in l_FileRows 
  16.  { 
  17.      // initialize record 
  18.      m_Record = Map(); 
  19.      m_Record.put("EmployeeID",""); 
  20.      m_Record.put("Name",""); 
  21.      m_Record.put("Address",""); 
  22.      m_Record.put("Territory",""); 
  23.      // 
  24.      // split values by commas 
  25.      l_FieldValues = r_Data.toList(","); 
  26.      if(l_FieldValues.size()>0) 
  27.      { 
  28.          m_Record.put("EmployeeID",l_FieldValues.get(0)); 
  29.      } 
  30.      if(l_FieldValues.size()>1) 
  31.      { 
  32.          m_Record.put("Name",l_FieldValues.get(1)); 
  33.      } 
  34.      if(l_FieldValues.size()>2) 
  35.      { 
  36.          m_Record.put("Address",l_FieldValues.get(2)); 
  37.      } 
  38.      if(l_FieldValues.size()>3) 
  39.      { 
  40.          m_Record.put("Territory",l_FieldValues.get(3)); 
  41.      } 
  42.      info m_Record.toString(); 
  43.  } 

Without a regex solution and the replacement back to commas then the above outputs:
copyraw 
{"EmployeeID":"1","Name":"Joel Lipman","Address":"Kings Castle","Territory":"England"}
{"EmployeeID":"11","Name":"General Dogsbody","Address":"\"Flat 8","Territory":" House Corner"}
{"EmployeeID":"64","Name":"Elephant Man","Address":"\"123 New Street","Territory":""}
{"EmployeeID":"New York City\"","Name":"USA","Address":"","Territory":""}
  1.  {"EmployeeID":"1","Name":"Joel Lipman","Address":"Kings Castle","Territory":"England"} 
  2.  {"EmployeeID":"11","Name":"General Dogsbody","Address":"\"Flat 8","Territory":" House Corner"} 
  3.  {"EmployeeID":"64","Name":"Elephant Man","Address":"\"123 New Street","Territory":""} 
  4.  {"EmployeeID":"New York City\"","Name":"USA","Address":"","Territory":""} 

Actions
  • Do you want to get rid of the annoying escaped double-quotes that no longer have any use in our data sample? Use a non-regex replaceall:
    copyraw 
    v_FormattedData = v_FormattedData.replaceAll("\"","",true);
    1.  v_FormattedData = v_FormattedData.replaceAll("\"","",true); 
  • If you are splitting your CSV rows by "\n", then you might want to consider that not all new lines/line breaks/carriage returns are at the end of a line.... for example, maybe your CSV contains multi-line fields; eg. where address street is on a different line to address city but in the same column... The following regex will replace new lines found in between 2 double-quotes and not at the end of the line:
    copyraw 
    v_FileContent = v_FileContent.replaceAll("(\"[^\"\n]*)\r?\n(?!(([^\"]*\"){2})*[^\"]*$)","$1|mySpecialNewLine|",false);
    1.  v_FileContent = v_FileContent.replaceAll("(\"[^\"\n]*)\r?\n(?!(([^\"]*\"){2})*[^\"]*$)","$1|mySpecialNewLine|",false); 
  • Replace all commas in between a pair of double-quotes: Well I couldn't find a single regex that can do this in one go but by applying the same regex to the same value (line 29), solved this for the sample data. If you are expecting a 3rd comma in the value, then apply the regex again.

The Final Solution:
The following code is a correction to the above and replaces any commas between two double-quotes with a custom string and then replaces this back to a comma when encountered. It will also replace any new lines found between two double-quotes with a single space (note how I've added a new line in the third row in the sample data):
copyraw 
// generate a sample CSV file 
 v_DataCSV = "1,Joel Lipman,Kings Castle,England\n"; 
 v_DataCSV = v_DataCSV + "11,General Dogsbody,\"Flat 8, House Corner, Street\",Brummieland\n"; 
 v_DataCSV = v_DataCSV + "64,Elephant Man,\"123 New Street,\nNew York City\",USA\n"; 
 f_CSVfile = v_DataCSV.toFile("test.csv"); 
 // 
 // read and process CSV file 
 v_FileContent = f_CSVfile.getFileContent(); 
 l_FileRows = List(); 
 if(!isBlank(v_FileContent)) 
 { 
     // replace the new line between two quotes with |mySpecialNewLine| 
	 v_FileContent = v_FileContent.replaceAll("(\"[^\"\n]*)[\r\n](?!(([^\"]*\"){2})*[^\"]*$)","$1|mySpecialNewLine|",false); 
     l_FileRows = v_FileContent.toList("\n"); 
 } 
 // loop through each row 
 for each r_Data in l_FileRows 
 { 
     // initialize record 
     m_Record = Map(); 
     m_Record.put("EmployeeID",""); 
     m_Record.put("Name",""); 
     m_Record.put("Address",""); 
     m_Record.put("Territory",""); 
     // 
     // replace the comma between two quotes with |mySpecialComma| 
	 v_FormattedData = r_Data.replaceAll("(\"[^\",]+)[,]([^\"]+\")","$1|mySpecialComma|$2",false); 
	 // again if there could be another comma in the value (repeat if more commas expected)
	 v_FormattedData = v_FormattedData.replaceAll("(\"[^\",]+)[,]([^\"]+\")","$1|mySpecialComma|$2",false); 
	 // replace any double-quotes 
     v_FormattedData = v_FormattedData.replaceAll("\"","",true); 
     // 
     // split values by commas 
     l_FieldValues = v_FormattedData.toList(","); 
     if(l_FieldValues.size()>0) 
     { 
         v_ThisColumnValue = l_FieldValues.get(0).trim(); 
         v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialComma|", ",", true); 
         v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialNewLine|", " ", true); 
         m_Record.put("EmployeeID",v_ThisColumnValue); 
     } 
     if(l_FieldValues.size()>1) 
     { 
         v_ThisColumnValue = l_FieldValues.get(1).trim(); 
         v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialComma|", ",", true); 
         v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialNewLine|", " ", true); 
         m_Record.put("Name",v_ThisColumnValue); 
     } 
     if(l_FieldValues.size()>2) 
     { 
         v_ThisColumnValue = l_FieldValues.get(2).trim(); 
         v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialComma|", ",", true); 
         v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialNewLine|", " ", true); 
         m_Record.put("Address",v_ThisColumnValue); 
     } 
     if(l_FieldValues.size()>3) 
     { 
         v_ThisColumnValue = l_FieldValues.get(3).trim(); 
         v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialComma|", ",", true); 
         v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialNewLine|", ", ", true); 
         m_Record.put("Territory",v_ThisColumnValue); 
     } 
     info m_Record.toString(); 
 }
  1.  // generate a sample CSV file 
  2.   v_DataCSV = "1,Joel Lipman,Kings Castle,England\n"; 
  3.   v_DataCSV = v_DataCSV + "11,General Dogsbody,\"Flat 8, House Corner, Street\",Brummieland\n"; 
  4.   v_DataCSV = v_DataCSV + "64,Elephant Man,\"123 New Street,\nNew York City\",USA\n"; 
  5.   f_CSVfile = v_DataCSV.toFile("test.csv"); 
  6.   // 
  7.   // read and process CSV file 
  8.   v_FileContent = f_CSVfile.getFileContent(); 
  9.   l_FileRows = List(); 
  10.   if(!isBlank(v_FileContent)) 
  11.   { 
  12.       // replace the new line between two quotes with |mySpecialNewLine| 
  13.       v_FileContent = v_FileContent.replaceAll("(\"[^\"\n]*)[\r\n](?!(([^\"]*\"){2})*[^\"]*$)","$1|mySpecialNewLine|",false); 
  14.       l_FileRows = v_FileContent.toList("\n"); 
  15.   } 
  16.   // loop through each row 
  17.   for each r_Data in l_FileRows 
  18.   { 
  19.       // initialize record 
  20.       m_Record = Map(); 
  21.       m_Record.put("EmployeeID",""); 
  22.       m_Record.put("Name",""); 
  23.       m_Record.put("Address",""); 
  24.       m_Record.put("Territory",""); 
  25.       // 
  26.       // replace the comma between two quotes with |mySpecialComma| 
  27.       v_FormattedData = r_Data.replaceAll("(\"[^\",]+)[,]([^\"]+\")","$1|mySpecialComma|$2",false); 
  28.       // again if there could be another comma in the value (repeat if more commas expected) 
  29.       v_FormattedData = v_FormattedData.replaceAll("(\"[^\",]+)[,]([^\"]+\")","$1|mySpecialComma|$2",false); 
  30.       // replace any double-quotes 
  31.       v_FormattedData = v_FormattedData.replaceAll("\"","",true); 
  32.       // 
  33.       // split values by commas 
  34.       l_FieldValues = v_FormattedData.toList(","); 
  35.       if(l_FieldValues.size()>0) 
  36.       { 
  37.           v_ThisColumnValue = l_FieldValues.get(0).trim(); 
  38.           v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialComma|", ",", true); 
  39.           v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialNewLine|", " ", true); 
  40.           m_Record.put("EmployeeID",v_ThisColumnValue); 
  41.       } 
  42.       if(l_FieldValues.size()>1) 
  43.       { 
  44.           v_ThisColumnValue = l_FieldValues.get(1).trim(); 
  45.           v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialComma|", ",", true); 
  46.           v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialNewLine|", " ", true); 
  47.           m_Record.put("Name",v_ThisColumnValue); 
  48.       } 
  49.       if(l_FieldValues.size()>2) 
  50.       { 
  51.           v_ThisColumnValue = l_FieldValues.get(2).trim(); 
  52.           v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialComma|", ",", true); 
  53.           v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialNewLine|", " ", true); 
  54.           m_Record.put("Address",v_ThisColumnValue); 
  55.       } 
  56.       if(l_FieldValues.size()>3) 
  57.       { 
  58.           v_ThisColumnValue = l_FieldValues.get(3).trim(); 
  59.           v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialComma|", ",", true); 
  60.           v_ThisColumnValue = v_ThisColumnValue.replaceAll("|mySpecialNewLine|", ", ", true); 
  61.           m_Record.put("Territory",v_ThisColumnValue); 
  62.       } 
  63.       info m_Record.toString(); 
  64.   } 
Yields:
copyraw 
{"EmployeeID":"1","Name":"Joel Lipman","Address":"Kings Castle","Territory":"England"}
{"EmployeeID":"11","Name":"General Dogsbody","Address":"Flat 8, House Corner, Street","Territory":"Brummieland"}
{"EmployeeID":"64","Name":"Elephant Man","Address":"123 New Street, New York City","Territory":"USA"}
  1.  {"EmployeeID":"1","Name":"Joel Lipman","Address":"Kings Castle","Territory":"England"} 
  2.  {"EmployeeID":"11","Name":"General Dogsbody","Address":"Flat 8, House Corner, Street","Territory":"Brummieland"} 
  3.  {"EmployeeID":"64","Name":"Elephant Man","Address":"123 New Street, New York City","Territory":"USA"} 

Source(s):
Category: Zoho :: Article: 759

Credit where Credit is Due:


Feel free to copy, redistribute and share this information. All that we ask is that you attribute credit and possibly even a link back to this website as it really helps in our search engine rankings.

Disclaimer: Please note that the information provided on this website is intended for informational purposes only and does not represent a warranty. The opinions expressed are those of the author only. We recommend testing any solutions in a development environment before implementing them in production. The articles are based on our good faith efforts and were current at the time of writing, reflecting our practical experience in a commercial setting.

Thank you for visiting and, as always, we hope this website was of some use to you!

Kind Regards,

Joel Lipman
www.joellipman.com

Related Articles

Joes Revolver Map

Joes Word Cloud

Accreditation

Badge - Certified Zoho Creator Associate
Badge - Certified Zoho Creator Associate

Donate & Support

If you like my content, and would like to support this sharing site, feel free to donate using a method below:

Paypal:
Donate to Joel Lipman via PayPal

Bitcoin:
Donate to Joel Lipman with Bitcoin bc1qf6elrdxc968h0k673l2djc9wrpazhqtxw8qqp4

Ethereum:
Donate to Joel Lipman with Ethereum 0xb038962F3809b425D661EF5D22294Cf45E02FebF
© 2024 Joel Lipman .com. All Rights Reserved.