C++ Tokenize vectors strings, no values between delimiters

[m6s]

1. After hours of researching, I used these snippets :

void Object::TokenizeLines(const string& str, vector<string>& tokens, const string& delimiters)
// Skip delimiters at beginning.
	string::size_type lastPos = str.find_first_not_of(delimiters, 0);
	// Find first "non-delimiter".
	string::size_type pos     = str.find_first_of(delimiters, lastPos);

	while (string::npos != pos || string::npos != lastPos) {
		// Found a token, add it to the vector.
		tokens.push_back( CheckWord( str, lastPos, pos - lastPos));
		// Skip delimiters.  Note the "not_of"
		lastPos = str.find_first_not_of(delimiters, pos);
		// Find next "non-delimiter"
		pos = str.find_first_of(delimiters, lastPos);
	}
}

for tokenizing large lines.
with a function call like this : string.Tokenize (str, tokens, "\n");
and because in each line I had things like : abc,de,,,f,g,,h
the previous lines were faulty. So I found this one :

void TokenizeWithComma(const string& str, vector<string>& tokens){
	const char* first = str.c_str();
	const char* last = str.c_str() + strlen(str.c_str());
	while (first != last) {
		const char* next = find(first, last, ',');
		tokens.push_back(string(first, next - first));
		first = min(next + 1, last);
	}
}

and I use it after the TokenizeLines to tokenise the words by passing a string and a vector. Both worked.

After spenting too many hours today for that, I have my homework done, but I am not sure what I did here...
Can someone (not novice likeme) give me more detailed view?

Also :
2. Why, the first function don't want to work with strings like a,bc,,d ?
3. I tryed also this for each word ( which are in vector ) :

for (w_iter = token_lines.begin(); w_iter != token_lines.end(); w_iter++) {
string ff = (*w_iter);
string::size_type loc = ff.find( "abc", 0 );
if( loc != string::npos ) { cout << "Found Omega at " << loc << endl;}
else {cout << "Didn't find Omega" << endl;}

But the code while is a working with a normal string ff("abc,ccc,cc" ), seems not to work with the declaration I had. Is the iterator's fault?
This drove me nuts, and made me in order to find patterns, devide a string with substr and have also cases for not making illegal measurements in the substr function ( which is of string again ).

I know it might be borring topic for most, I appreciate your help...
Thank you

[weaknessforcats]

But the code while is a working with a normal string ff("abc,ccc,cc" ),

string ff("abc,ccc,cc" ) creates a strring object.

Your function has a string& argument so you can use that string object as a argument.

This ("abc,ccc,cc ") is a C-string. A C-string cannot be used a string& because it's not a string object.

Finally, I like your last solution. You should never be use the string::c_str() method unless your function absolutely requires a C-string. Considerinf that the C string library is deprecated in C++, there should little call for this.

[m6s]

string ff("abc,ccc,cc" ) creates a strring object.

Your function has a string& argument so you can use that string object as a argument.

This ("abc,ccc,cc ") is a C-string. A C-string cannot be used a string& because it's not a string object.

Finally, I like your last solution. You should never be use the string::c_str() method unless your function absolutely requires a C-string. Considerinf that the C string library is deprecated in C++, there should little call for this.

Thank you for your answer, and the good word :-)
I had tried even string temp = *iter, but that didn't work also. I assume by this way it should load a string object and tokenize it, right?
because temp was then passed to the function.But didn't work either.
Finally, how could I make the TokenizeCommas without a c_str().
Build the whole string as char arrays? Or is any other choice in order to stick as much as it can close to C++?

[weaknessforcats]

Why can't you use find() to locate the next comma??

[m6s]

:-) I Don't understand me either!!!
I have a 64bit, can that be the problem?
Just go through yourself if you like it and find it interesting...
I know that didn't result.

In other words, can you make an iterator from words ( if from a file even better) which will be like a,b,c,,,d,f,g ....so on... (did you notice the three commas-->2 spaces?

And then use with just C++ code not C, I mean the first Tokenize function.
What's your result? Oh, assign each iterator to a string and then tokenize it.
For me it was disaster...

[weaknessforcats]

In other words, can you make an iterator from words ( if from a file even better) which will be like a,b,c,,,d,f,g ....so on... (did you notice the three commas-->2 spaces?

It's not the string that's your problem. Its your parsing logic.

There is an article in the C/C++ HowTos on the State Design Pattern and inside that article is how tro constrcuct a tokenizer to break a string into individual words. Complete with code. You might read that article.

[m6s]

Ok, thank you for your close support on this, I am going to check this article too...