INN InterNet News

Computer Center, CS, NCTU 2 Introduction to INN (1)  Originally written by Rich Salz ISC took over development in  INN is a complete Usenet system innd – NNTP server, main program, inbound receiver  Exchange news articles with peers  innfeed: outbound, feed articles to peers nnrpd – NNTP reader server Separate readers and peers  For readers: readers.conf  For peers: incoming.conf 、 innfeed.conf 、 newsfeeds A lot of components…

Computer Center, CS, NCTU 3 Introduction to INN (2)

Computer Center, CS, NCTU 4 Introduction to INN (3)  Main configuration inn.conf  Components innd – main daemon for peers (and users)  incoming.conf innfeed – NNTP feeder  innfeed.conf 、 newsfeeds nnrpd – for users  readers.conf Newsgroups database  active 、 active.times 、 newsgroups 、 history Overview database  buffindexed.conf 、 ovdb.conf 、 overview.fmt Spool  storage.conf 、 cycbuff.conf

Computer Center, CS, NCTU 5 Introduction to INN (4)  Specific functions Article and history expiration  expire.ctl Handling of control messages  control.ctl Submission addresses for moderated groups  moderators Cron jobs  news.daily Summery of INN log files  innreports.conf Others  …

INN Installation

Computer Center, CS, NCTU 7 Install INN (1)  In FreeBSD ports news/inn: INN INN was made EOL by the release of Mar-25 - INN v2.5.2, ISC is pleased to announce a new bug fixed version of INN 2.5  How to install INN 2.5.2? Tarball: ftp://ftp.isc.org/isc/inn/inn tar.gzftp://ftp.isc.org/isc/inn/inn tar.gz Dependencies: “Requirements” in  C compiler gcc: built-in  Perl and MIME::Parser: lang/perl5.8+ 、 mail/p5-MIME-Tools 、 graphics/p5-GD  GNU make: devel/gmake  GnuPG: security/gnupg  Berkeley DB: databases/db47+  Yacc implementation: devel/bison

Computer Center, CS, NCTU 8 Install INN (2)    Set up account ‘news’  Set up the home directory for ‘news’ # mkdir –p /home/news && chown news:news /home/news You *CANNOT* chown -R news:news after installing. Some bins are SUID root.  Login as ‘news’ su - news / sudo -u news … rlogin –l news localhost  Enable rlogind via inetd, and set ~news/.rhosts –localhost username news:*:8:8::0:0:News Subsystem:/home/news:/bin/tcsh

Computer Center, CS, NCTU 9 Install INN (3)  The source Fetch the tarball and put it in /tmp by fetch or wget(ftp/wget) Extract it: tar xzf inn tar.gz  Configure./configure --help  --prefix=$HOME  --with-http-dir=$HOME/public_html  --enable-largefiles cannot be used with --enable-tagged-hash  --enable-keywords  --with-perl  --with-berkeleydb=/usr/local You can copy some environment variables from ports/news/inn  make -n CONFIG_DONE_INN=1 /usr/ports/news/inn/work/.configure_done.inn._usr_local  CC, CFLAGS, CXXFLAGS, LDFLAGS

Computer Center, CS, NCTU 10 Install INN (4)  After configure  Modify Makefile.global Please check the following files before running make, to ensure that everything was set correctly. Makefile.global include/config.h include/inn/options.h include/inn/paths.h innfeed/innfeed.h ## Berkeley DB support. If this support is configured, anything linking ## against libstorage also needs to link against DB_LDFLAGS and DB_LIBS. DB_CPPFLAGS = -I/usr/local/include/db47 DB_LDFLAGS = -L/usr/local/lib DB_LIBS = -ldb-4.7 –lz DBM_CPPFLAGS = -I/usr/local/include/db47 DBM_LIBS = -L/usr/local/lib -ldb-4.7

Computer Center, CS, NCTU 11 Install INN (5)  Compile INN $ gmake (make)  Install INN Login as ‘root’ # gmake (make) install  To upgrade INN # gmake (make) update  upgrade binaries, scripts, and manual pages, but not config files  Be sure to configure INN with the same options that was used previously.  Finally Create mail alias for (which is defined in --with-news-master) Set.tcshrc setenv PATH $HOME/bin:$PATH setenv MANPATH $HOME/share/man:`manpath`

Computer Center, CS, NCTU 12 Install INN (6)  Set up syslog In /etc/syslog.conf  $ cd ~/log && touch news.crit news.err news.notice  # cd /var/log && ln –s ~news/log news  # /etc/rc.d/syslogd reload  Log rotate Done by news.daily, which is triggered by cron # uncomment these if you're running inn news.crit /var/log/news/news.crit news.err /var/log/news/news.err news.notice /var/log/news/news.notice

Computer Center, CS, NCTU 13 Install INN (7)  Set up cron jobs crontab for ‘news’ $ crontab –e  news.daily If you're using any non-CNFS storage methods, add delayrm to the above option list for news.daily.  ~news/log/expire.rm  expirerm file  rnews Processing spooled messages to the server again 0 3 * * * ~news/bin/news.daily expireover lowmark 0 3 * * * ~news/bin/rnews -U

Computer Center, CS, NCTU 14 Install INN (8)  Copy start-up script from news/inn/files/innd.sh.in Put it in /usr/local/etc/rc.d/innd Modify ‘%VAR%’s  %PREFIX%  ~news (/home/news 、 /usr/local/news 、 …)  %DBDIR$$  ~news/db  %NEWSBASE%  ~news # chmod 555 /usr/local/etc/rc.d/innd  Modify /etc/rc.conf  A lot of configurations remain… innd_enable=“YES”

INN Databases ~news/db/

Computer Center, CS, NCTU 16 active (1)  List the newsgroups carried by INN. Maintained using ctlinnd or control messages Edit it directly: active.times, overview database  Format Each newsgroup is listed only once Each line specifies one newsgroup Four fields separated by a space:   cs.test y name: junk 、 control* high is the highest article number ever seen low is not guaranteed to be accurate, and should only be taken to be a hint.  Normally updated nightly as part of the expire process If the lowest article number is greater than the highest article number, then there are no articles in the newsgroup.

Computer Center, CS, NCTU 17 active (2)  Format status:  y: allow posting and feeding  m: moderated, all posting must be approved  n: allow feeding only  j: fed articles are filed in junk group  x: disallow posting and feeding  =foo.bar: articles are filed in the newsgroup foo.bar  Get a close to complete newsgroups database file ftp://ftp.isc.org/pub/usenet/CONFIG/ active 、 newsgroups  tw.bbs.*  Big5 newsgroups for tw.bbs.*

Computer Center, CS, NCTU 18 active.times  Provide a chronological record of when newsgroups were created on the local server  Format Three fields   cs.test time: the time it was created, expressed as Unix timestamp  date -r creator: plain text intended to describe the entity that created the newsgroup  This field should be encoded in UTF-8  Usually the address of the creator

Computer Center, CS, NCTU 19 newsgroups  Contain a list of newsgroups and its short description It is not necessary that all the groups carried by the news server be listed in the newsgroups file It is also not necessary that all the groups listed in the newsgroups file be carried by the news server  If you use ctlinnd newgroup to manually create a group, only the active file is updated edit newsgroups file to add a short description for the created group  Format Each line consists of two fields separated by at least one tab  \t  cs.computer-center[ 系計中公告 ] (Moderated)  Preferred format: The total line length should be at most 79 columns. The description should start with a capital and not be more than 55 characters long  using UTF-8 for non-ASCII characters in description is recommended

Computer Center, CS, NCTU 20 history (1)  Keep a record of current and recently expired articles  Format [ ]\t \t [E7516D3A69051BC9712D093D52A4C67B] ~- ~ Each line corresponds to one article Hash: ASCII representation of the hash of the Message-ID header date: three sub-fields separated by a ‘~’, each contains Unix timestamp  1 st sub-field: arrival time of the article  2 nd sub-field: ‘Expires:’ header or ‘-’  3 rd sub-field: ‘Date:’ header, record of the posting time token: a token of the article.  This field is empty if the article has been expired  Use sm utility to retrieve the article

Computer Center, CS, NCTU 21 history (2)  grephistory Query the INN history database via Message-ID Output the token if the article is still on the news server By use of sm to retrieve the article  makehistory Initialize or rebuild INN history database Also can be used to rebuild the overview database  makehistory –FOx  Be sure to stop innd and delete or zero out the existing database

INN Configurations ~news/etc/

Computer Center, CS, NCTU 23 Article Storage Format  The supported storage formats tradspool  Articles are stored as individual text files, named by the article number, and divided up into directories based on the newsgroups name  article in cs.test would be stored as cs/test/12345 timehash  Articles are stored as individual text files, and divided into directories based on the arrival time. timecaf  Similar to timehash, but multiple articles are put in the same file cnfs  Articles are stored in pre-configured buffer files Advantages and Disadvantages  INN can support all four at the same time

Computer Center, CS, NCTU 24 Overview Storage Mechanism  Overview is for readers  Three overview mechanisms tradindexed  Uses two files per newsgroup, one containing the overview data and one containing the index  Fast for readers, but slow to write to buffindexed  Stores overview data and index information into buffers, which are preconfigured files defined in buffindexed.conf, similar to CNFS buffers  Slower for readers ovdb  Stores overview data in a Berkeley DB database  Fast and robust, but may require more disk space

Computer Center, CS, NCTU 25 Configration outline  Access controls Peers: incoming.conf 、 newsfeeds 、 innfeed.conf Readers: readers.conf  Storage cycbuff.conf 、 storage.conf  Overview ovdb.conf 、 buffindexed.conf 、 overview.fmt  Other controls Expiration: expire.ctl Moderators list: moderators INN report: innreport.conf Default subscriptions for a new client: subscriptions …

Computer Center, CS, NCTU 26 inn.conf (1)  The primary general configuration file for all INN programs Most likely to need setting  organization:“College of Computer Science, NCTU” –It will be filled in the ‘Organization:’ header for posts  ovmethod:ovdb –The overview mechanism General Settings  server:csnews2.cs.nctu.edu.tw –Default NNTP server for nnrpd, actsync, nntpget, getlist Feed Configuration  pathalias:news.cs.nctu.edu.tw –News servers within a particular organization to add a common identity Article Storage  extraoverviewadvertised:[ ]  Extraoverviewhidden:[ ] –See inn.conf(5)

Computer Center, CS, NCTU 27 inn.conf (2) Reading  noreader:true –For separately running innd and nnrpd Posting  nntpauthsender:true –Add ‘Sender:’ header to local posts containing the identity assigned by readers.conf Monitoring  doinnwatch:false –Disable the innwatch Logging  docnfsstat:true –Start cnfsstat when innd is started  logcycles:3 –How many old logs scanlogs keeps

Computer Center, CS, NCTU 28 incoming.conf  Specify who are permitted to connect to your host and feed it articles Connections from hosts not listed in this file will be rejected or be handed off to nnrpd and checked against the access restrictions in readers.conf  Three types of entries: key/value, peer, group streaming: true # streaming allowed by default max-connections: 8 # per feed group cs { max-connections: 16 patterns: peer CSBBS { hostname: "bbs.cs.nctu.edu.tw, " } peer FREEBSD { hostname: "freebsd.cs.nctu.edu.tw, " patterns: "!*,freebsd.*,mailing.*" } } group peering { max-connections: 16 patterns: peer NCTUPEER { hostname: "news-peer.nctu.edu.tw, " max-connections: 8 } }

Computer Center, CS, NCTU 29 newsfeeds (1)  Determine how incoming articles are redistributed to your peers Organized as a series of feed entries Each entry is composed of four fields separated by ‘:’ 1 st field: /,,…  If you don't want a feed to receive articles from a certain source, then … 2 nd field: /  wildmat pattern matching syntax –These are simple wildcard matches using the ‘*’ as the wildcard character –wildmat patterns can be specified in a comma-separated list –The last pattern in the line that matches the group name is used –Patterns beginning with ! mean to exclude groups matching that pattern »*,!comp.*,comp.os.* –wildmat patterns also support "poison" patterns (patterns starting vs. “misc.*,!misc.bar” »For a article crossposted between misc.foo and misc.bar »For a article posted only to misc.bar

Computer Center, CS, NCTU 30 newsfeeds (2) 3 rd field: comma-separated list of flags  th field: multi-purpose parameter, meaning depends on the flags in 3 rd field  Feeding peers via innfeed  Site settings ## Uncomment if you're using innfeed. ## Add "-y" as an option to innfeed to use the name of each feed as the ## name of the host to feed articles to; without "-y" an innfeed.conf ## file is needed. # innfeed funnel master. innfeed!\ :!*\ :Tc,Wnm*:/home/news/bin/innfeed -y news.uu.net/uunet\ :*,!junk,!control,!control.*/!foo\ :Tm:innfeed!

Computer Center, CS, NCTU 31 newsfeeds (3)  Other examples News overview used when “useoverchan: true” in inn.conf CS Archive-SQLite Mirror nctu.talk and tw.bbs.campus.nctu # News overview. overview!:*:Tc,WnteO:/news/bin/overchan archive_sqlite_cs!\ :Tc,Wn:/news/bin/archive-sqlite.pl cs # mirror mirrorpost!/mirrorpost,netnews.cs.nctu.edu.tw\ :!*,nctu.talk,tw.bbs.campus.nctu\ :Tc,Wf:/news/bin/mirrorpost.pl nctu.talk tw.bbs.campus.nctu

Computer Center, CS, NCTU 32 innfeed.conf  Used to control the innfeed program  Three types of entries: key/value, peer, group (as incoming.conf) peer { # body } is the same as the site name in ‘newsfeeds’  If you do not specify -y in ‘innfeed!’ entry in newsfeeds, then …  In newsfeeds  In innfeed.conf NCTUCSBBS/bbs.cs.nctu.edu.tw,BS2\ :!*,cs.*,tw.bbs.*,twbbs.*,taiwan.*,nctu.*,mailing.*, :Tm:innfeed! group BBS { max-connections: 2 streaming: false port-number: 7777 peer NCTUCSBBS { ip-name: bbs.cs.nctu.edu.tw } }

Computer Center, CS, NCTU 33 readers.conf (1)  Specify access control for nnrpd  Two types of entries: parameter/value, configuration groups :  No way to continue a line on the next line, and no way to have a value longer than about 8,180 characters Configuration groups  auth: provides mechanisms to establish the identity of the user  access: given the user's identity, what that user is permitted to do  readers.conf is a two-step process Put all of the auth groups first, and all of the access groups below, last match A user identity, as established by an auth group, looks like an address “ When matching access groups the last matching one in the file is used to determine the user's permissions 

Computer Center, CS, NCTU 34 readers.conf (2)  auth group Generate the user identity like   nnrpdauthsender auth "EVERYWHERE" { hosts: "*" default: " " default-domain: "EVERYWHERE" } auth "NCTUCSCC" { hosts: "bsd*.cs.nctu.edu.tw, linux*.cs.nctu.edu.tw, sun*.cs.nctu.edu.tw" default-domain: "NCTUCSCC" res: "ident -t" }

Computer Center, CS, NCTU 35 readers.conf (3)  access group newsgroups vs. read + post access: R(read) 、 P(Post) 、 I(IHAVE) 、 A(Approver) 、 N(NEWNEWS) 、 L(post to non-local posting newsgroups, [jnx] in active) access "READING_LIMIT" { users: access: "R" read: "!*,cs.help,freebsd.*,gmane.*,mailing.*" } access "CS" { users: access: "RP" newsgroups: "*,!control*,!cs.prsystem,!cs.cc.*,!csie.cc.*,!cis.sysadmin.*,!csie.sysadmin.*,! cis.computer-center.staff,!cis.syslog.*,!cis.bbs.admin.*,!junk" }

Computer Center, CS, NCTU 36 cycbuff.conf  Define the cyclical buffers that make up the storage pools for CNFS To use any sysbuff larger than 2GB: --enable-largefiles  Formats cycbuffupdate: refreshinterval: cycbuff: : :  No longer than 7 characters in, and than 63 in  is the length in KB (2GB= )  cycbuff:BIG80:/home/news/cycbuffs/BIG80: cycbuff:BIG81:/home/news/cycbuffs/BIG81:  dd if=/dev/zero of=/home/news/cycbuffs/BIG80 bs=1k count= metacycbuff: : [,,...][: ]  Mode: INTERLEAVE(default) or SEQUENTIAL  metacycbuff:BIG8:BIG80,BIG81

Computer Center, CS, NCTU 37 storage.conf (1)  Contain the rules used to assign articles to different storage methods  Format : tradspool 、 timehash 、 timecaf 、 cnfs 、 trash : a unique number in to identify the storage  Used in expire.ctl, timehash, and timecaf : only for CNFS to specify the metacycbuff name method { newsgroups: class: size: [, ] expires: [, ] options: exactmatch: }

Computer Center, CS, NCTU 38 storage.conf (2)  An example method tradspool { newsgroups: cs.cc.* class: 0 } method cnfs { newsgroups: cs.*, eecsep.*,nctu.* class: 1 size: 0,8191 options: NCTUCS } method cnfs { newsgroups: comp.*,humanities.*,misc.*,news.*,rec.*,sci.*,soc.*,talk.* class: 2 options: BIG8 }

Computer Center, CS, NCTU 39 overview  ovdb.conf Parameters for tuning ovdb, no need to modify  buffinedxed.conf : : entries (similar to cycbuff) is unique in , is in KB  overview.fmt No longer used by INN since v2.5 Use extraoverviewadvertised and extraoverviewhidden in inn.conf

Computer Center, CS, NCTU 40 expire.ctl (1)  Default configuration file for expire and expireover, define how long History entries for expired or rejected articles are remembered  /remember/: Articles stored on the server are retained  : : : : –if ‘groupbaseexpiry: true‘ in inn.conf (default) –Last match – : ! not permitted, and only a single expression – : M(moderated) 、 U(unmoderated) 、 A(all) 、 X(delete from all groups)  : : : –if ‘groupbaseexpiry: false‘ in inn.conf – : defined in storage.conf, ‘*’ to specify a default for all classes ,, : decimal number of days, fraction is allowed or ‘never’ , : come into play with an ‘Expire:’ header  : used as the expiration period for most articles

Computer Center, CS, NCTU 41 expire.ctl (2)  An example # Keep expired article history for 11 days, matching artcutoff plus one. /remember/:11 # Most articles stay for two weeks, ignoring Expires: headers. *:A:14:14:14 # Accept Expires: headers in moderated groups for up to a year and # retain moderated groups for a bit longer. *:M:1:30:365 # Keep local groups for a long time and local project groups forever. example.*:A:1:90:90 example.project.*:A:never:never:never

Computer Center, CS, NCTU 42 moderators  Determine submission addresses for moderated newsgroups  Format : First match for  More specific patterns should be listed before general patterns : a simple address  At most one ‘%s’ may occur and be replaced by the name of the newsgroup, with all ‘.’ in the name changed to ‘-’ ## CS moderated newsgroup ## Public hierarchies with exceptions. ## Direct all other public hierarchies to the master moderator database.

Computer Center, CS, NCTU 43 innreport.conf  Configuration file for innreport  Default parameters section default:  htmltrue;  title"Daily Usenet report for mynews";  footer"Local contact:  html_css_url "innreport.css";  A lot of options for report content and appearance

Control Messages

Computer Center, CS, NCTU 45 Control Messages (1)  Cancels are handled internally by INN  Others are processed by controlchan (newsfeeds) The actions of controlchan are determined by control.ctl  ftp://ftp.isc.org/pub/usenet/CONFIG/control.ctl ftp://ftp.isc.org/pub/usenet/CONFIG/control.ctl newgroup 、 rmgroup 、 checkgroups  control* control control.cancel control.checkgroups control.newgroup control.rmgroup

Computer Center, CS, NCTU 46 Control Messages (2)  Authenticate control messages based on the ‘From:’ header Obviously perilous and control messages are widely forged Many hierarchies sign all of their control messages with PGP controlchan knows how to do this (using pgpverify) without additional configuration  PGP key import ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS $ gpg --import PGPKEYS $ ln -s ~/.gnupg ~/etc/pgp

Computer Center, CS, NCTU 47 Control Messages (3) Path: csnews2!csnews.cs.nctu.edu.tw!news.cs.nctu.edu.tw!ctu-peer!ctu-gate! news.nctu.edu.tw!newsfeed.berkeley.edu!ucberkeley!solaris.cc.vt.edu!news.vt.edu!guardian.oit.duke.edu!news.glorb.com!news2.glorb.com!usenet.stanford. edu!usenet-its.stanford.edu!bounce-back From: Newsgroups: sci.physics.acoustics Subject: cmsg newgroup sci.physics.acoustics Control: newgroup sci.physics.acoustics Approved: Date: Sat, 15 May :00: Message-ID: X-PGP-Sig: 2.6.3a Subject,Control,Message-ID,Date,From,Sender \011iQCVAwUBS+79QsJdOtO4janBAQGNsAP7BAj2Vl4LS2RoIZmYTfHc9GFg/n Fve8Hj \011iOYqpW+WiF7pI6JaNzPaeS/Y3Dh7G9HRjjhuYbQ/+bTUHWeDItRFBbGVe 3t+yXEv \011TbQ/NqWqMug/OKujGsOs9wy1HQKZBErDIfxCf+XD6JenRccrBPAZoCMdS FieztZe \011vFU9B4Az+ew= \011=d9AE Xref: csnews.cs.nctu.edu.tw control.newgroup: For your newsgroups file: sci.physics.acoustics\011Topics in acoustics and vibrations.

Computer Center, CS, NCTU 48 Control Messages (4)  control.ctl Last match Format  : : : : all, newgroup, rmgroup, checkgroups, … : match the ‘From:’ header :  doit 、 drop 、 log 、 mail 、 verify-pgp_userid Encoding of newgroup and checkgroups control messages  /encoding/:*:nctu.*:big5  /encoding/:*:tw.*:big5  /encoding/:*:fido7.*:koi8-r  /encoding/:*:fido.*:utf-8  /localencoding/:utf-8 Checkgroups  /maxdocheckgroups/:*:*:10

More on INN control and operations

Computer Center, CS, NCTU 50 Start and Stop  Before starting, check the configurations $ inncheck -a -v -f -pedantic -perm  /usr/local/etc/rc.d/innd su -fm news -c "/home/news/bin/rc.news start" su -fm news -c "/home/news/bin/rc.news stop"  If you run nnrpd separately su -fm news -c "/home/news/bin/nnrpd –D" su -fm news -c "killall nnrpd"

Computer Center, CS, NCTU 51 Reload the configurations  Using ctlinnd  When INN is running To get your incoming feeds working  Edit incoming.conf  $ ctlinnd reload incoming.conf 'reason‘ To get your outgoing feeds working  Edit newsfeeds and corresponding innfeed.conf or nntpsend.ctl  ctlinnd reload newsfeeds 'reason‘

Computer Center, CS, NCTU 52 Create/Remove Newsgroups  Use control messages  Use ctlinnd Create  ctlinnd newgroup [ status [creator] ]  Add the description in db/newsgroups Remove  ctlinnd rmgroup  Optional remove the description in db/newsgroups

Computer Center, CS, NCTU 53 news.daily  Perform a number of important Usenet administrative functions Producing a status report with innstat Removing old news articles (with expirerm if the delayrm keyword is specified) Purging the overview database with expireover if the corresponding eponym keyword is specified Processing log files and rotating the archived log files with scanlogs Processing innfeed dropped files with procbatch Renumbering the active file with ctlinnd Rebuilding the history file with expire Removing any old socket files found in the pathrun directory Collecting all the output and mailing it

Computer Center, CS, NCTU 54 mailpost  A program to feed an message into a newsgroup Read a properly formatted message from stdin Feed it to inews for posting to a news server  Normally, mailpost is run via mail aliases freebsd-acpi: "|/home/news/bin/mail-post mailing.freebsd.acpi"

Computer Center, CS, NCTU 55 Further reading  INN 2.5 Documentation  INN changes and upgrade information  INN 2.x FAQ  Cleanfeed