I have been evaluating the excellent and super fast Similarity Analyser by Red Hill Consulting to generate a duplicate lines reports from Rails 1.2.1 source code (excluding the tests) with a threshold of 5 lines.

May I insist on the threshold parameter: this parameter must be >=5, hence identical blocks of lines whose length official features page of Simian.

Update 23/01/2007: There was a bug in simian 2.2.12 which was considering lines containing the "end" keyword as duplicates. Hopefully, Simmon Harris fixed it in couple of days and released the version 2.2.13.

I have written a bit of code to generate the following html report from simian text report, I will publish as soon as it I have put into a gem and document it.

Here is the report generated on the 23th of January 2007:


Similarity Analyser 2.2.13 - http://www.redhillconsulting.com.au/products/simian/index.html

Copyright (c) 2003-07 RedHill Consulting Pty. Ltd. All rights reserved.

Simian is not free unless used solely for non-commercial or evaluation purposes.

{failOnDuplication=true, ignoreCharacterCase=true, ignoreCurlyBraces=true, ignoreIdentifierCase=true, ignoreModifiers=true, ignoreStringCase=true, threshold=5}

Loading (recursively) *[^t][^e][^s][^t].rb from /home/jeanmichel/ruby/projects/duplicate-lines-report/tmp

Found 5 duplicate lines in the following files:

   Between lines 745 and 751 in activerecord/lib/active_record/connection_adapters/frontbase_adapter.rb

   Between lines 90 and 96 in activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb

Found 5 duplicate lines in the following files:

   Between lines 688 and 692 in railties/lib/initializer.rb

   Between lines 37 and 41 in activesupport/lib/active_support/ordered_options.rb

Found 5 duplicate lines in the following files:

   Between lines 218 and 224 in actionmailer/lib/action_mailer/vendor/tmail/encode.rb

   Between lines 147 and 153 in actionmailer/lib/action_mailer/vendor/tmail/encode.rb

Found 5 duplicate lines in the following files:

   Between lines 418 and 424 in actionpack/lib/action_controller/assertions/selector_assertions.rb

   Between lines 293 and 301 in actionpack/lib/action_controller/assertions/selector_assertions.rb

Found 5 duplicate lines in the following files:

   Between lines 2 and 7 in railties/lib/commands/servers/webrick.rb

   Between lines 9 and 14 in railties/lib/commands/servers/mongrel.rb

Found 5 duplicate lines in the following files:

   Between lines 95 and 100 in activerecord/lib/active_record/deprecated_associations.rb

   Between lines 35 and 40 in activerecord/lib/active_record/deprecated_associations.rb

Found 5 duplicate lines in the following files:

   Between lines 43 and 47 in actionwebservice/examples/metaWeblog/apis/blogger_api.rb

   Between lines 34 and 38 in actionwebservice/examples/metaWeblog/apis/blogger_api.rb

Found 5 duplicate lines in the following files:

   Between lines 77 and 81 in actionwebservice/examples/googlesearch/delegated/google_search_service.rb

   Between lines 27 and 31 in actionwebservice/examples/googlesearch/direct/search_controller.rb

   Between lines 26 and 30 in actionwebservice/examples/googlesearch/autoloading/google_search_controller.rb

Found 5 duplicate lines in the following files:

   Between lines 178 and 183 in actionmailer/lib/action_mailer/vendor/tmail/parser.rb

   Between lines 127 and 132 in actionmailer/lib/action_mailer/vendor/tmail/parser.rb

Found 5 duplicate lines in the following files:

   Between lines 676 and 683 in railties/lib/initializer.rb

   Between lines 6 and 13 in activesupport/lib/active_support/ordered_options.rb

Found 5 duplicate lines in the following files:

   Between lines 679 and 683 in actionmailer/lib/action_mailer/vendor/tmail/parser.rb

   Between lines 678 and 682 in actionmailer/lib/action_mailer/vendor/tmail/parser.rb

Found 5 duplicate lines in the following files:

   Between lines 1375 and 1381 in actionmailer/lib/action_mailer/vendor/tmail/parser.rb

   Between lines 1359 and 1365 in actionmailer/lib/action_mailer/vendor/tmail/parser.rb

Found 5 duplicate lines in the following files:

   Between lines 68 and 72 in activerecord/lib/active_record/connection_adapters/openbase_adapter.rb

   Between lines 117 and 121 in activerecord/lib/active_record/connection_adapters/sqlite_adapter.rb

Found 5 duplicate lines in the following files:

   Between lines 25 and 31 in railties/lib/rails_generator.rb

   Between lines 29 and 33 in actionpack/lib/action_controller.rb

   Between lines 29 and 33 in activerecord/lib/active_record.rb

Found 5 duplicate lines in the following files:

   Between lines 202 and 213 in activerecord/lib/active_record/connection_adapters/mysql_adapter.rb

   Between lines 180 and 193 in activerecord/lib/active_record/connection_adapters/oracle_adapter.rb

Found 5 duplicate lines in the following files:

   Between lines 831 and 837 in railties/lib/commands/plugin.rb

   Between lines 809 and 815 in railties/lib/commands/plugin.rb

   Between lines 774 and 780 in railties/lib/commands/plugin.rb

   Between lines 603 and 609 in railties/lib/commands/plugin.rb

   Between lines 573 and 579 in railties/lib/commands/plugin.rb

   Between lines 546 and 552 in railties/lib/commands/plugin.rb

Found 5 duplicate lines in the following files:

   Between lines 5 and 16 in railties/lib/commands/server.rb

   Between lines 78 and 89 in railties/lib/commands/process/spawner.rb

Found 5 duplicate lines in the following files:

   Between lines 226 and 235 in actionmailer/lib/action_mailer/vendor/tmail/parser.rb

   Between lines 208 and 218 in actionmailer/lib/action_mailer/vendor/tmail/parser.rb

Found 5 duplicate lines in the following files:

   Between lines 2 and 7 in railties/lib/rails/version.rb

   Between lines 2 and 7 in actionwebservice/lib/action_web_service/version.rb

Found 5 duplicate lines in the following files:

   Between lines 12 and 18 in activerecord/lib/active_record/connection_adapters/openbase_adapter.rb

   Between lines 70 and 75 in activerecord/lib/active_record/connection_adapters/mysql_adapter.rb

Found 5 duplicate lines in the following files:

   Between lines 499 and 503 in activerecord/lib/active_record/connection_adapters/firebird_adapter.rb

   Between lines 478 and 482 in activerecord/lib/active_record/connection_adapters/firebird_adapter.rb

Found 5 duplicate lines in the following files:

   Between lines 317 and 321 in actionpack/lib/action_controller/vendor/html-scanner/html/selector.rb

   Between lines 296 and 300 in actionpack/lib/action_controller/vendor/html-scanner/html/selector.rb

   Between lines 277 and 281 in actionpack/lib/action_controller/vendor/html-scanner/html/selector.rb

Found 5 duplicate lines in the following files:

   Between lines 1 and 7 in activerecord/test/fixtures/migrations/1_people_have_last_names.rb

   Between lines 1 and 7 in activerecord/test/fixtures/migrations_with_duplicate/1_people_have_last_names.rb

   Between lines 1 and 7 in activerecord/test/fixtures/migrations_with_missing_versions/1_people_have_last_names.rb

Found 5 duplicate lines in the following files:

   Between lines 680 and 685 in activerecord/lib/active_record/vendor/simple.rb

   Between lines 627 and 631 in activerecord/lib/active_record/vendor/simple.rb

Found 6 duplicate lines in the following files:

   Between lines 16 and 25 in activesupport/lib/active_support/core_ext/time/conversions.rb

   Between lines 11 and 21 in activesupport/lib/active_support/core_ext/date/conversions.rb

Found 6 duplicate lines in the following files:

   Between lines 197 and 202 in activerecord/lib/active_record/connection_adapters/sqlserver_adapter.rb

   Between lines 134 and 139 in activerecord/lib/active_record/connection_adapters/sybase_adapter.rb

Found 6 duplicate lines in the following files:

   Between lines 284 and 291 in actionmailer/lib/action_mailer/vendor/tmail/header.rb

   Between lines 344 and 349 in actionmailer/lib/action_mailer/vendor/tmail/header.rb

Found 6 duplicate lines in the following files:

   Between lines 267 and 272 in activerecord/lib/active_record/validations.rb

   Between lines 825 and 830 in activerecord/lib/active_record/validations.rb

Found 6 duplicate lines in the following files:

   Between lines 277 and 285 in actionpack/lib/action_controller/vendor/html-scanner/html/selector.rb

   Between lines 296 and 304 in actionpack/lib/action_controller/vendor/html-scanner/html/selector.rb

Found 6 duplicate lines in the following files:

   Between lines 358 and 365 in actionmailer/lib/action_mailer/vendor/tmail/header.rb

   Between lines 498 and 505 in actionmailer/lib/action_mailer/vendor/tmail/header.rb

Found 6 duplicate lines in the following files:

   Between lines 175 and 185 in activerecord/lib/active_record/connection_adapters/oracle_adapter.rb

   Between lines 108 and 118 in activerecord/lib/active_record/connection_adapters/openbase_adapter.rb

Found 7 duplicate lines in the following files:

   Between lines 408 and 414 in actionwebservice/test/abstract_dispatcher.rb

   Between lines 398 and 404 in actionwebservice/test/abstract_dispatcher.rb

Found 7 duplicate lines in the following files:

   Between lines 518 and 530 in actionmailer/lib/action_mailer/vendor/tmail/header.rb

   Between lines 221 and 233 in actionmailer/lib/action_mailer/vendor/tmail/header.rb

Found 7 duplicate lines in the following files:

   Between lines 76 and 82 in actionmailer/lib/action_mailer/vendor/tmail/scanner_r.rb

   Between lines 68 and 74 in actionmailer/lib/action_mailer/vendor/tmail/scanner_r.rb

Found 7 duplicate lines in the following files:

   Between lines 164 and 176 in activerecord/lib/active_record/connection_adapters/openbase_adapter.rb

   Between lines 264 and 276 in activerecord/lib/active_record/connection_adapters/mysql_adapter.rb

Found 7 duplicate lines in the following files:

   Between lines 1 and 10 in activerecord/test/fixtures/migrations/3_innocent_jointable.rb

   Between lines 1 and 10 in activerecord/test/fixtures/migrations_with_duplicate/3_innocent_jointable.rb

   Between lines 1 and 10 in activerecord/test/fixtures/migrations_with_missing_versions/4_innocent_jointable.rb

Found 7 duplicate lines in the following files:

   Between lines 181 and 192 in actionpack/lib/action_controller/assertions/selector_assertions.rb

   Between lines 60 and 67 in actionpack/lib/action_controller/assertions/selector_assertions.rb

Found 7 duplicate lines in the following files:

   Between lines 204 and 213 in actionmailer/lib/action_mailer/vendor/tmail/parser.rb

   Between lines 148 and 157 in actionmailer/lib/action_mailer/vendor/tmail/parser.rb

Found 7 duplicate lines in the following files:

   Between lines 279 and 285 in activerecord/lib/active_record/connection_adapters/frontbase_adapter.rb

   Between lines 94 and 100 in activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb

Found 7 duplicate lines in the following files:

   Between lines 123 and 132 in actionpack/lib/action_controller/assertions/response_assertions.rb

   Between lines 69 and 78 in actionpack/lib/action_controller/assertions/routing_assertions.rb

Found 7 duplicate lines in the following files:

   Between lines 18 and 25 in railties/lib/rails_generator/generators/components/web_service/web_service_generator.rb

   Between lines 14 and 21 in railties/lib/rails_generator/generators/components/controller/controller_generator.rb

Found 7 duplicate lines in the following files:

   Between lines 506 and 512 in activerecord/lib/active_record/connection_adapters/sybase_adapter.rb

   Between lines 534 and 540 in activerecord/lib/active_record/connection_adapters/sqlserver_adapter.rb

Found 8 duplicate lines in the following files:

   Between lines 58 and 69 in railties/lib/rails_generator/generators/components/scaffold/scaffold_generator.rb

   Between lines 18 and 29 in railties/lib/rails_generator/generators/components/resource/resource_generator.rb

   Between lines 18 and 29 in railties/lib/rails_generator/generators/components/scaffold_resource/scaffold_resource_generator.rb

Found 8 duplicate lines in the following files:

   Between lines 37 and 49 in actionmailer/lib/action_mailer/vendor/tmail/stringio.rb

   Between lines 178 and 190 in actionmailer/lib/action_mailer/vendor/tmail/stringio.rb

Found 8 duplicate lines in the following files:

   Between lines 24 and 33 in activerecord/lib/active_record.rb

   Between lines 24 and 33 in actionpack/lib/action_controller.rb

Found 8 duplicate lines in the following files:

   Between lines 366 and 374 in activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb

   Between lines 219 and 227 in activerecord/lib/active_record/connection_adapters/sqlserver_adapter.rb

Found 8 duplicate lines in the following files:

   Between lines 5 and 18 in activesupport/lib/active_support/core_ext/module/attribute_accessors.rb

   Between lines 23 and 36 in activesupport/lib/active_support/core_ext/module/attribute_accessors.rb

Found 8 duplicate lines in the following files:

   Between lines 5 and 18 in activesupport/lib/active_support/core_ext/class/attribute_accessors.rb

   Between lines 23 and 36 in activesupport/lib/active_support/core_ext/class/attribute_accessors.rb

Found 9 duplicate lines in the following files:

   Between lines 6 and 14 in railties/lib/commands/process/spawner.rb

   Between lines 3 and 11 in railties/lib/commands/process/spinner.rb

   Between lines 4 and 12 in activesupport/lib/active_support/core_ext/kernel/daemonizing.rb

Found 10 duplicate lines in the following files:

   Between lines 77 and 88 in actionpack/lib/action_controller/assertions/selector_assertions.rb

   Between lines 195 and 210 in actionpack/lib/action_controller/assertions/selector_assertions.rb

Found 10 duplicate lines in the following files:

   Between lines 740 and 757 in actionmailer/lib/action_mailer/vendor/tmail/header.rb

   Between lines 836 and 853 in actionmailer/lib/action_mailer/vendor/tmail/header.rb

Found 10 duplicate lines in the following files:

   Between lines 458 and 469 in activerecord/lib/active_record/connection_adapters/frontbase_adapter.rb

   Between lines 482 and 493 in activerecord/lib/active_record/connection_adapters/frontbase_adapter.rb

Found 11 duplicate lines in the following files:

   Between lines 64 and 75 in actionwebservice/examples/googlesearch/delegated/google_search_service.rb

   Between lines 14 and 25 in actionwebservice/examples/googlesearch/direct/search_controller.rb

   Between lines 13 and 24 in actionwebservice/examples/googlesearch/autoloading/google_search_controller.rb

Found 12 duplicate lines in the following files:

   Between lines 40 and 52 in railties/lib/rails_generator/generators/components/scaffold/scaffold_generator.rb

   Between lines 2 and 14 in railties/lib/rails_generator/generators/components/resource/resource_generator.rb

   Between lines 2 and 14 in railties/lib/rails_generator/generators/components/scaffold_resource/scaffold_resource_generator.rb

Found 12 duplicate lines in the following files:

   Between lines 84 and 106 in actionwebservice/examples/googlesearch/delegated/google_search_service.rb

   Between lines 33 and 55 in actionwebservice/examples/googlesearch/autoloading/google_search_controller.rb

   Between lines 34 and 56 in actionwebservice/examples/googlesearch/direct/search_controller.rb

Found 19 duplicate lines in the following files:

   Between lines 53 and 81 in railties/lib/rails_generator/generators/components/scaffold_resource/scaffold_resource_generator.rb

   Between lines 42 and 69 in railties/lib/rails_generator/generators/components/resource/resource_generator.rb

Found 29 duplicate lines in the following files:

   Between lines 2 and 40 in railties/lib/rails_generator/generators/components/scaffold_resource/scaffold_resource_generator.rb

   Between lines 2 and 40 in railties/lib/rails_generator/generators/components/resource/resource_generator.rb

Found 37 duplicate lines in the following files:

   Between lines 3 and 56 in actionwebservice/examples/googlesearch/direct/search_controller.rb

   Between lines 2 and 55 in actionwebservice/examples/googlesearch/autoloading/google_search_controller.rb

Found 38 duplicate lines in the following files:

   Between lines 2 and 81 in activesupport/lib/active_support/binding_of_caller.rb

   Between lines 2 and 82 in railties/lib/binding_of_caller.rb

Found 41 duplicate lines in the following files:

   Between lines 1 and 49 in actionwebservice/examples/googlesearch/delegated/google_search_service.rb

   Between lines 1 and 49 in actionwebservice/examples/googlesearch/direct/google_search_api.rb

   Between lines 1 and 49 in actionwebservice/examples/googlesearch/autoloading/google_search_api.rb

Found 1074 duplicate lines in 121 blocks in 60 files

Processed a total of 23454 significant (46787 raw) lines in 375 files

Processing time: 1.347sec



Now the debate is opened about what to do with these reports !!!