Shell script to check the unique numbers in huge data

namishtiwari · #1 (**permalink**) 08-10-07, 03:43

Friends,
I have to write a shell script,the description is----

i Have to check the uniqueness of the numbers in a file.
A file is containing 200thousand tickets and a ticket have 15 numbers in asecending order.And there is a strip that is having 6 tickets that means 90 numbers.I have to write a shell script like this----- No two tickets are common in 200thousand tickets and no two numers are repeating in a strip.
I would appericiate your help.

If you want some more info i am pasting it here.
---------------------------------------------------
Each ticket strip will have six tickets.
A strip is a group of 6 tickets. Every possible number from 1 through to 90 will appear on each full strip.
Each ticket will have 9 columns, and 3 rows. Hence a ticket strip will effectively have 9 columns and 18 rows.
Column 1 will have numbers between 1-9, 2 will have 10-19, so on till 8th column having 70-79; the only exception is the last column (9th column) having 80-90.
In each row there will be only 5 columns generated, the other 4 will be blank.
If there are more than 1 value in a column within a ticket, then they should be in ascending order.

namishtiwari · #2 (**permalink**) 08-10-07, 04:56

Here is the format of the file.

insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (100,1,'071400004200007480000021364450660000001623 390000007784',1);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (101,1,'050000004055627100001028330000680087080000 004959007988',2);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (102,1,'001522300054600000060000004156007090091926 310000610000',3);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (103,1,'000000004652677685011329384700000000001700 000057697886',4);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (104,1,'020020340000007581001825004351640000030027 370000650083',5);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (105,1,'041100004553007200000024324800630082001200 350058007389',6);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (106,2,'000022340057007685071027004800620000000000 360059687789',1);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (107,2,'021221004600000090051500310050007100080024 004700617800',2);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (108,2,'001300334553000084040026000056657500001800 394900660086',3);

namishtiwari · #3 (**permalink**) 08-10-07, 05:30

Here is the format of the data---

insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (100,1,'071400004200007480000021364450660000001623 390000007784',1);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (101,1,'050000004055627100001028330000680087080000 004959007988',2);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (102,1,'001522300054600000060000004156007090091926 310000610000',3);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (103,1,'000000004652677685011329384700000000001700 000057697886',4);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (104,1,'020020340000007581001825004351640000030027 370000650083',5);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (105,1,'041100004553007200000024324800630082001200 350058007389',6);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (106,2,'000022340057007685071027004800620000000000 360059687789',1);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (107,2,'021221004600000090051500310050007100080024 004700617800',2);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (108,2,'001300334553000084040026000056657500001800 394900660086',3);

The requirement is like this---

1 strip is having six tickets containing all 90 numbers.
This number should be unique.

gvee · #4 (**permalink**) 08-10-07, 05:49

Would this be useful?

Code:

IF EXISTS (SELECT id FROM sysobjects WHERE id = object_id('MyTable')) BEGIN
	DROP TABLE MyTable
END

DECLARE @x int, @RepeatCount int, @Value char(9)
SET @x = 0
SET @RepeatCount = 100

CREATE TABLE MyTable(SomeField char(9))

WHILE @x < @RepeatCount BEGIN
	SET @Value = convert(varchar,replicate(0,9 - Len(@x))) + convert(varchar,@x)
	INSERT INTO MyTable(SomeField) VALUES (@Value)
	SET @x = @x + 1
END

SELECT * FROM MyTable
DROP TABLE MyTable

namishtiwari · #5 (**permalink**) 08-10-07, 05:55

Thanks georgev,
I will try this one.Can you suggest something in shell script.

I appericiate your help.

gvee · #6 (**permalink**) 08-10-07, 05:57

Well I don't really understand your requirements, and I don't know shell...
I was merely commenting on the SQL you were using and having a guess.

Run the script and let me know if it gives you the required effect.

stolze · #7 (**permalink**) 08-10-07, 06:52

Quote:

Originally Posted by namishtiwari

Here is the format of the data---

insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (100,1,'071400004200007480000021364450660000001623 390000007784',1);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (101,1,'050000004055627100001028330000680087080000 004959007988',2);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (102,1,'001522300054600000060000004156007090091926 310000610000',3);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (103,1,'000000004652677685011329384700000000001700 000057697886',4);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (104,1,'020020340000007581001825004351640000030027 370000650083',5);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (105,1,'041100004553007200000024324800630082001200 350058007389',6);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (106,2,'000022340057007685071027004800620000000000 360059687789',1);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (107,2,'021221004600000090051500310050007100080024 004700617800',2);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (108,2,'001300334553000084040026000056657500001800 394900660086',3);

The requirement is like this---

1 strip is having six tickets containing all 90 numbers.
This number should be unique.

Do you have the data in a file or in such an SQL script? If it is a SQL script, does your requirement include to evaluate the data inside a database?

If it is in a flat file, you could build a shell script that extracts the portions of the data you are interested in, combine these portions into a single line for each record, and then run "sort" or "sort -u". Comparing the line count of "sort" and "sort -u" tells you if there were duplicates found and eliminated by "sort -u".

namishtiwari · #8 (**permalink**) 08-10-07, 08:25

Thanks for the help experts.
jsut now the they change the requirement,
i have pasted the requirement as a new thread by my name.
Please help me in that.

Thanks in advance.

Internet Services

Web Development

Digital Marketing

Consumer Tech