Problem with Comment Approval emails

[perlnerd]

1.0 Beta2 downloaded and installed today. FreeBSD 5 Stable php 4.3.11, apache-1.3.33

The final line in the email is corrupted:

Approve comment -- http://www.domain.com/serendipity/index ... ��������ѵ�((���)M�є�9��́�́�ݕɕ����M�ɕ������)Q������Ё������ɽչ����ԁ�����͔��Ёѽ��)
�������Ѐ����輽��乽ɜ�Ѽ��������Ё��ܸ

Here's the full text of the message (with some identifying info removed)
***********************************************************
X-Account-Key: account1
X-UIDL: hD'#!4]C"!SP0!!(H&#!
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Received: from k(k)
by 3(8.11.1/8.11.1) with ESMTP id k2RKQWM95554
for <>; Mon, 27 Mar 2006 14:26:32 -0600 (CST)
(envelope-from www@)
Received: from k (localhost [127.0.0.1])
by k (8.13.3/8.13.1) with ESMTP id k2RKQWl8029158
for <t>; Mon, 27 Mar 2006 14:26:32 -0600 (CST)
(envelope-from www@)
Received: (from www@localhost)
by k (8.13.3/8.13.1/Submit) id k2RKQWII029157;
Mon, 27 Mar 2006 14:26:32 -0600 (CST)
(envelope-from www)
Date: Mon, 27 Mar 2006 14:26:32 -0600 (CST)
From: www user <www@>
To: my address
Subject: [Site News] [Requires review] New comment posted to Me again...........
Reply-To: "Dave Davis" <>
X-Mailer: Serendipity/1.0-beta2
X-Engine: PHP/4.3.11
Message-ID: <fcf120a99661ac6a7631d78015628c36@www.***.com>
MIME-Version: 1.0
Precedence: bulk
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by k id k2RKQWM95554
X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on k
X-Spam-Level:
X-Spam-Status: No, score=-5.9 required=3.8 tests=ALL_TRUSTED,BAYES_00
autolearn=ham version=3.0.4
X-UIDL: hD'#!4]C"!SP0!!(H&#!

A new comment has been posted on your blog "Site News", to the entry entitled "Me again...........".
Link to entry: http://www.domain.com/serendipity/index ... ......html

Requires review: Yes
User IP-address: 65.95.241.165
User Name: Dave Davis
User Email:
User Homepage:

Comments:
THisiis a test

----
You have the following options available:
Warning: This comment needs approval before it will be displayed
View comment -- http://www.domain.com/serendipity/index ... ...html#c4
Delete comment -- http://www.domain.com/serendipity/index ... ......html
Approve comment -- http://www.domain.com/serendipity/index ... мм̵5TµE]E¥¸¸¸¸¸¸¸¸¸¸¸¸¹¡Ñµ°((´´@)M¥ÑTA9UÝÌA¥ÌAÁ½ÝUÉUPAIäAMUÉU¹Q¥Á¥Ñä¸)Q¡TAIUÍÐAI±½\AEɽչP°Aå½ÔAME¸AÕÍTA¥ÐAѽ¼¸)M¡UM¬A½ÕÐ@ñ¡ÑÑÀè¼½Ìåä¹½É\øAѼAY¥¹PA½ÕÐA¡½Ü¸

******************************************************

I changed the address assocaitted with my account so that the messages would go to a host not running SpamAssassin and The messages are still garbled at the end:

*************************************************
X-Mailer: Serendipity/1.0-beta2
X-Engine: PHP/4.3.11
Message-ID: <a941bd99f10c2e69fef53f9c33510e96@www.pethobbyist.com>
MIME-Version: 1.0
Precedence: bulk
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by k id k2RLKDg29804

A new comment has been posted on your blog "Site News", to the entry entitled "Some
Advertiser News.....".
Link to entry: http://www.domain.com/serendipity/index ... ......html

Requires review: Yes
User IP-address: 65.95.241.165
User Name: clint
User Email: clint@klcb.ca
User Homepage:

Comments:
thios is etst. not a very good oje th0ough

----
You have the following options available:
Warning: This comment needs approval before it will be displayed
View comment -- http://www.domain.com/serendipity/index ... ...html#c7
Delete comment -- http://www.domain.com/serendipity/index ... ......html
Appsù½Ù”??½µµ•¹Ð€´´?¡ÑÑÀè¼½ÝÝܹÁ•Ñ¡½‰‰å¥Íй?½´½Í•É•¹‘¥Á¥Ñä½¥¹‘•à¹Á¡Àü½…ÁÁɽٔ½?½µµ•¹Ð¼Ü¼ÔµM½µ”µ‘Ù•Éѥ͕ȵ9•Ý̸¸¸¸¸¹¡Ñµ°((´´€)M¥Ñ”?9•ÝÌ?¥Ì?Á½Ý•É•??‰ä?M•É•¹‘¥Á¥Ñä¸)Q¡”?‰•ÍÐ?‰±½œ?…ɽչ?°?å½Ô??…¸?ÕÍ”?¥Ð?ѽ¼¸)
¡•?¬?½ÕЀñ¡ÑÑÀè¼½Ìå乽ɜø?Ѽ?™¥¹??½ÕÐ?¡½Ü¸

**********************************************************

Any ideas?

Thanks
Perlnerd

[judebert]

That's a corrupted string, all right. I'm still running alpha1, myself, so I can't confirm your results directly. However, you're the only one on the forums with the problem, so it's possible there was a corruption in the particular version you downloaded, or a problem when you uploaded the files.

I'd try re-uploading, and if that didn't work I'd try a daily that was one day older or newer. If that didn't work, I'd post on the forums calling me an idiot and asking for more detailed help.

[perlnerd]

I'd try re-uploading, and if that didn't work I'd try a daily that was one day older or newer. If that didn't work, I'd post on the forums calling me an idiot and asking for more detailed help.

Ok, I downloaded a new copy of 1.0 BETA2 to my server, untarred it and copied it over the other install. I still get corrupted text in the email notifications.

I'll try a snapshot from a different date tomorrow and let you know.

Thanks for the help.

[perlnerd]

I installed and upgraded to the snapshot from March 14 and went through the upgrade (to 1.1-alpha1).

Emails are still showing gibberish characters at the end.

Any help you can offer would be appreciatted.

Thanks

*****************
X-Account-Key: account1
X-UIDL: MJ+"!o$K!!"!H"!7h7"!
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Received: from m (m)
by k (8.11.1/8.11.1) with ESMTP id k2SFqvM50533
for <>Tue, 28 Mar 2006 09:52:57 -0600 (CST)
(envelope-from www@)
Received: from .com (localhost [127.0.0.1])
by .com (8.13.3/8.13.1) with ESMTP id k2SFqvZc070330
for <techservices@>; Tue, 28 Mar 2006 09:52:57 -0600 (CST)
(envelope-from www@)
Received: (from www@localhost)
by .com (8.13.3/8.13.1/Submit) id k2SFqvum070329;
Tue, 28 Mar 2006 09:52:57 -0600 (CST)
(envelope-from www)
Date: Tue, 28 Mar 2006 09:52:57 -0600 (CST)
To: techservices
Subject: [Site News] [Requires review] New comment posted to Testing media files
From: "" <DoNotReply@>
Reply-To: "" <@.ca>
X-Mailer: Serendipity/1.1-alpha1
X-Engine: PHP/4.3.11
Message-ID: <6db39a3b60e445df0f868de890bda00b@www.pethobbyist.com>
MIME-Version: 1.0
Precedence: bulk
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by .com id k2SFqvM50533
X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on .com
X-Spam-Level:
X-Spam-Status: No, score=-5.9 required=3.8 tests=ALL_TRUSTED,AWL,BAYES_00
autolearn=ham version=3.0.4
X-UIDL: MJ+"!o$K!!"!H"!7h7"!

A new comment has been posted on your blog "Site News", to the entry entitled "Testing media files".
Link to entry: http://www..com/serendipity/index.php?/archives/6-Testing-media-files.html

Requires review: Yes
User IP-address: 65.95.125.236
User Name: Name
User Email: email@email.com
User Homepage:

Comments:
This is atest to see if the emails are still screwed up.

----
You have the following options available:
Warning: This comment needs approval before it will be displayed
View comment -- http://www.domain.com/serendipity/index ... s.html#c12
Delete comment -- http://www.domain.com/serendipity/index ... files.html
Appsù½Ù½µµ¹Ð´´¡ÑÑÀè¼½ÝÝܹÁÑ¡½å¥Íй½´½Íɹ¥Á¥Ñä½¥¹à¹Á¡Àü½ÁÁɽٽ½µµ¹Ð¼ÄȼصQÍÑ¥¹µµ¥µ¥±Ì¹¡Ñµ°((´´)M¥Ñ9ÝÌ¥ÌÁ½ÝÉäMɹ¥Á¥Ñä¸)Q¡Íб½É½Õ¹°å½Ô¸ÕÍ¥ÐÑ¡¬½ÕÐñ¡ÑÑÀè¼½Ìåä¹½ÉøѼ¥¹½ÕС½Ü¸



*****************

[judebert]

Well, at least it's consistent.

That sure looks like a bug. Especially since it's the same corruption every time. Hmmm...

It's possible that serendipity_rewriteURL() is messing up, but that's unlikely because it's also used for the previous line. It's possible serendipity_makeFilename() is messing up, but unlikely for the same reason.

I assume your language is English. Check the language file for the APPROVE_COMMENT and SIGNATURE constants, and make sure they're OK. If it is, open include/functions_comments.inc.php and find the code like this (hopefully around line 634 somewhere):

Code: Select all

        $text = sprintf(A_NEW_COMMENT_BLAHBLAH, $serendipity['blogTitle'], $title)
              . "\n" . LINK_TO_ENTRY . ': ' . $entryURI
              . "\n"
              . "\n" . REQUIRES_REVIEW         . ': ' . (($moderate_comment) ? YES : NO) . (isset($serendipity['moderate_reason']) ? ' (' . $serendipity['moderate_reason'] . ')' : '')
              . "\n" . USER . ' ' . IP_ADDRESS . ': ' . $_SERVER['REMOTE_ADDR']
              . "\n" . USER . ' ' . NAME       . ': ' . $fromName
              . "\n" . USER . ' ' . EMAIL      . ': ' . $fromEmail
              . "\n" . USER . ' ' . HOMEPAGE   . ': ' . $fromUrl
              . "\n"
              . "\n" . COMMENTS                . ': '
              . "\n" . strip_tags($comment)
              . "\n"
              . "\n" . '----'
              . "\n" . YOU_HAVE_THESE_OPTIONS
              . (($moderate_comment) ? "\n" . str_repeat(' ', 2) . THIS_COMMENT_NEEDS_REVIEW : '')
              . "\n" . str_repeat(' ', 3) . str_pad(VIEW_COMMENT,  15) . ' -- '. $entryURI .'#c'. $comment_id
              . "\n" . str_repeat(' ', 3) . str_pad(DELETE_COMMENT,  15) . ' -- '. $deleteURI
              . (($moderate_comment) ? "\n" . str_repeat(' ', 3) . str_pad(APPROVE_COMMENT, 15) . ' -- '. $approveURI : '');

Try adding this on the next line:

Code: Select all

$text .= "\nThis is the end of the email.\n";

I'm thinking your last line is getting corrupted in your mailer. If so, this should at least leave the URL decent. It's possible the terminating \n is all you need. If you like, you can even experiment with different ways of composing that $text variable until you find something you like better.

Let me know if you have any problems with this.

[perlnerd]

Hi

Thanks for the additional help.

I tried what you suggested, but the problem persists.

**************************
You have the following options available:
Warning: This comment needs approval before it will be displayed
View comment -- http://www.pethobbyist.com/sitenews/ind ... ..html#c14
Delete comment -- http://www.pethobbyist.com/sitenews/ind ... �ѡ��������(((���)M�є�9��́�́�ݕɕ����M�ɕ������)Q������Ё������ɽչ����ԁ�����͔��Ёѽ��)
�������Ѐ����輽��乽ɜ�Ѽ��������Ё��ܸ
***************************
I did go an modify this in function_comments.inc:

return serendipity_sendMail($to, $subject, $text, $fromEmail, null, $fromName);

to

Mail($to, $subject, $text, "From: $fromEmail");

Lo and behold, the message comes thorugh just fine (even the "This is the end of the emai" line added earlier.

********************
A new comment has been posted on your blog "Site News", to the entry entitled "Some Advertiser News.....".
Link to entry: http://www.pethobbyist.com/sitenews/ind ... ......html

Requires review: Yes
User IP-address: 65.95.125.236
User Name: Clint Gilders
User Email: dave@davedavis.com
User Homepage:

Comments:
efefefd wf wef ww fwefwef2w wfwfewf

----
You have the following options available:
Warning: This comment needs approval before it will be displayed
View comment -- http://www.pethobbyist.com/sitenews/ind ... ..html#c18
Delete comment -- http://www.pethobbyist.com/sitenews/ind ... ......html
Approve comment -- http://www.pethobbyist.com/sitenews/ind ... ......html
This is the end of the email.

*********************

So, what might culprit be in serendipity_sendMail()?

I tried removing the 8bit and 64 bit encoding lines and all I got was a mostly blank email with a bit of the gibberish:

*******************
w�r��z{aj��zzh�ם�|����Ҋ׍{���-�秶���ح���J���ޮجz�^������������m����

********************

I did notice than in the heads it says: "Content-Transfer-Encoding: 8bit", but serendipity_sendMail supposedly adds "Content-Transfer-Encoding: quoted-printable" or "Content-Transfer-Encoding: base64", and not what I'm seeing.

I modified the function to tell me what it is using in the email:

if (LANG_CHARSET == 'UTF-8') {
if (function_exists('imap_8bit')) {
$maildata['headers'][] = 'Content-Transfer-Encoding: quoted-printable';
$maildata['message'] = "This is 8bit" . $maildata['message'];
$maildata['message'] = str_replace("=\r\n", "", imap_8bit($maildata['message']));

} else {
$maildata['headers'][] = 'Content-Transfer-Encoding: base64';
$maildata['message'] = "This is base 64 " . $maildata['message'];
$maildata['message'] = base64_encode($maildata['message']);

}
}

And I see "This is base 64" at the beginning of the message.

Hope some of this is useful. Let me know if I can provide you with anything else.

Thanks!

[judebert]

I'm just glad it's working for you -- and that you have some debugging skills. It could easily drive me crazy.

Frankly, it sounds like the base64 encoding is going crazy. An engineer would say that's impossible. As a programmer, I know we have to deal with what actually happens, impossible or not.

If you're interested in helping out, the next logical step is to examine the input and output of the base64_encode function. Since we can't put it in the mail (because it's what gets corrupted anyway), and we don't want to confuse actual users by printing it on the blog, we could put it in a file. I'd use PHP's syslog function to write them both to a file I could access on my webserver. (Heck, I'd even name it log.html or something.)

Right now, though, I'm leaning toward the X-MIME-AutoConverted header. It indicates that someone did a conversion behind our backs. I'm pretty confident that PHP encodes base64 correctly, so I'll bet some utility in the email chain is mucking things up.[/url]

[judebert]

Of course, AFTER I post the help request, THEN I find a possible solution.

Just before the base64_encode() function, try adding set_magic_quotes_runtime(0). It's possible the string is being interpreted with a base64 slash as an escape character.

Just a thought.

[perlnerd]

Ok, the set_magic_quotes_runtime(0) didn't do anything.

php's syslog seems to only allow a certain number of characters so the log entries are truncated. I figure that would render the data useless.

I whipped up a little function to write the strings to a file. I then did a base64_decode on it and it decoded just fine. See http://www.pethobbyist.com/sitenews/include/decode.php

We send lots of mail using php, and have never seen any problems before. But, we just use php's mail function and don't get into any of the encoding that you do.

Any thoughts on what I might be able to do?

[judebert]

For now, I'd leave it with the mail() function, just like you did. It seemed to work as expected. Will you be able to help us out later, when we get to examining this problem in greater detail? I'd like to know we could duplicate it.

Hmmm... why couldn't we just set the encoding to 8-bit and leave it?

[perlnerd]

For now, I'd leave it with the mail() function, just like you did. It seemed to work as expected. Will you be able to help us out later, when we get to examining this problem in greater detail? I'd like to know we could duplicate it.

Hmmm... why couldn't we just set the encoding to 8-bit and leave it?

I'd be glad to help in anyway I can. Overall we're very happy with the software, and will probably be going live with a dozen or more installs shortly.

Thanks again for all your assistance.

[garvinhicking]

Hi!

First off, the problem seems that your MTA is doing Autoconversion.

Serendipity sends the mail in base64encoding, but your MTA does X-MIME-Autoconversion to 8bit.

8bit is problematic in some mail clients (Outlook express) an not RFC conform. It works in some cases, but not all.

That's why we stick with quoted-printable (if imap_8bit function available or base64 encoding for sending UTF-8 E-Mails. Otherwise, persian characters in UTF-8 mode will definitely get mangled.

Now, the extra characters seem to come from your MTA, sadly. Can you see what causes this auto-conversion, and either disable it, or fix it?

Best regard,s
Garvin

[perlnerd]

Considering I'm using a stock install of sendmail on FreeBSD and sucessfully send hunders (if not thousands) of messages a day from perl, shell , and php scripts I wouldn't think there would be anything to fix at the MTA level.

Rather, I'd like to see serendipity work out of the box.

Thanks

[perlnerd]

From what I can gather "Content-Transfer-Encoding: 8bit';" is still a valid " Content-Transfer-Encoding Header Field". I have added that header to the emails and disabled the imap_8bit and base64_encode section and the mail comes through nicely.

It also seems to work if I set it to 7bit. I'm not sure which it should be in this case.

I guess what I might like to see is a configuration switch where the encoding can be turned off. So if someone else has the same problem there is an easy way to fix it.

Thanks.

[garvinhicking]

Our plugin routines allow to hook into a plugin that sets mail headers there easily.

Giving config options for this is a hard thing to do, because with the current way everything should work flawlessly. Actually, it would work, if your MTA wouldn't be autoconverting anything. I only know debian linux, and it doesn't do this neither with postfix nor sendmail nor exim.

And as I mentioned, using 8bit is not an option for several email clients out there...

I understand that you want to make s9y work out of the box, but if something on the MTA side is changing what s9y wants to send, we're a bit at loss there...

Regards,
Garvin